KR102094509B1 - Method of modifying image including photographing restricted element, device and system for performing the same - Google Patents

Abstract

The user device according to an embodiment of the present invention recognizes a person included in an image and a storage unit in which data is stored, and if the recognized person is not related to the user by using the data stored in the storage unit, And a controller that corrects a part of the image corresponding to the recognized person so that the recognized person cannot be identified.

Description

{Method of modifying image including photographing restricted element, device and system for performing the same}

The present invention relates to a method for correcting an image when a photograph or video photographed on a user device includes an element for which photographing is restricted, and a device and a system for performing the same.

In the case of taking a picture or a video through a shooting device such as a camera or a camcorder, a person unknown to the photographer may be included in the captured image. However, irrespective of the intention of the person, the photographed person may not want to be exposed to another person's intention, and in some cases, a privacy protection issue may arise. However, it is difficult for the photographer to ensure that the person who does not know is not included in the video every time he shoots.

If, among the characters included in the captured image, a person whose shooting is restricted is classified and image processing is performed so that the person cannot be identified, the photographer can easily shoot the same way and protect the privacy of the subject. You can expect the effect.

In particular, in recent years, as devices, such as smartphones or tablets, which are always carried by individuals at all times support the shooting function, there is an increased possibility of being exposed to other people's shooting, and thus, a countermeasure is needed.

On the other hand, it may be a specific object or a specific location, as well as others who do not know that the shooting needs to be restricted to the individual. Accordingly, when an object or location in which an image is captured included in an image captured by an individual device is restricted, it is necessary to perform image processing so that it cannot be identified from the image.

It is intended to provide a method of correcting an image when an element for which shooting is restricted is included in an image photographed on a personal device.

The user device according to an embodiment of the present invention for solving the above technical problem recognizes a storage unit in which data is stored, and a person included in an image, and uses the data stored in the storage unit to recognize the recognized person. And a control unit that corrects a part of the image corresponding to the recognized person so that identification of the recognized person is not possible if the user is not related to each other.

In this case, the user device may further include a photographing unit for photographing an image.

At this time, the control unit may be characterized in that it determines whether the location where the image is photographed is a location where photographing is restricted, and when a photographing is restricted, a part of the image corresponding to the recognized person is corrected.

In this case, when the data stored in the storage unit is an address book, the controller may determine that the user and the recognized person are not related to each other if the recognized person does not exist in the address book.

At this time, when the data stored in the storage unit is at least one picture, the controller determines that the user and the recognized person are not related to each other if the recognized person does not exist in the at least one picture. It can be characterized as.

At this time, the control unit may be characterized by determining whether the user and the recognized person are related to each other by analyzing the user's SNS (Social Networking Service) record using the data stored in the storage unit. .

At this time, the controller may access the database of an external server to determine whether the location where the image is captured is a location where photography is restricted, and if it is determined that the location is restricted, a part of the image may be corrected have.

At this time, the control unit may be characterized in that, when recognizing a shooting-restricted object from the image, the location where the image is captured is determined as a location where the shooting is restricted.

In this case, when it is determined that a part of the image needs to be corrected, the controller performs at least one of mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing on a part of the video. You can.

A system for modifying an image according to another embodiment of the present invention for solving the above technical problem recognizes a user device transmitting an image to a server and requesting image processing, and a person included in the image received from the user device And if the recognized person is not related to the user, modify a part of the image corresponding to the recognized person so that identification of the recognized person is impossible, and transmit a modified image to the user device. Includes.

At this time, the server is characterized in that using the pre-stored data to determine whether the location where the image is photographed is a location where photography is restricted, and if a location where the image is restricted is to modify a part of the image corresponding to the recognized person. Can be done with

At this time, the server receives the time and location at which the image was taken from the user device, and determines whether or not shooting of the recognized person is restricted at the received time and location using pre-stored data. can do.

The user device according to another embodiment of the present invention for solving the above technical problem, a photographing unit for photographing an image, and when photographing is performed, transmits a photographing notification to a device of a photographed person, and determines whether photographing is allowed from the photographed device. It includes a receiving communication unit, and a control unit that corrects the captured image according to whether the photographing is allowed.

A method of correcting an image according to an embodiment of the present invention for solving the technical problem includes: recognizing a person included in the image, determining whether the recognized person is related to the user, and And if the recognized person is not related to the user, modifying a part of the image corresponding to the recognized person so that identification of the recognized person is impossible.

At this time, the step of determining whether or not the recognized person is related to the user may further include determining whether a location where the image is photographed is a location where photographing is restricted, and correcting a part of the image. The step may be a step of correcting a part of the image when the location where the image is photographed is a location where photographing is restricted.

At this time, the step of determining whether the recognized person is related to the user may include: if the recognized person does not exist in the address book stored in the user device, the user and the recognized person are not related to each other. It may be characterized in that the step of judging.

At this time, the step of determining whether the recognized person is related to the user may include: if the recognized person does not exist in at least one picture stored in the user device, the user and the recognized person are related to each other. It may be characterized in that the step of determining that there is no.

At this time, the step of determining whether or not the recognized person is related to the user is analyzed by analyzing the user's SNS (Social Networking Service) record using the data stored in the user device. It may be characterized in that it is a step of determining whether the characters are related to each other.

At this time, the step of correcting a part of the image may include performing at least one of mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing on a part of the video.

A user device according to an embodiment of the present invention for solving the above technical problem includes: a photographing unit for photographing an image; And a recognition of at least one of a person, an object, or a location where the image is captured, included in the captured image, and a relationship between a user of the device and the recognized character, the recognized object, or the recognized location. It may include a control unit for performing the correction of the image based on at least one.

At this time, it further includes a storage unit in which data is stored, and the control unit determines whether the user and the recognized person are related to each other by using the data stored in the storage unit and corrects the image when it is determined that it is not related. You can.

In addition, at this time, an address book is stored in the storage unit, and if the recognized person does not exist in the address book, the controller may determine that the user and the recognized person are not related to each other.

Alternatively, at least one picture is stored in the storage unit, and if the recognized person does not exist in the at least one picture, the controller may determine that the user and the recognized person are not related to each other.

Alternatively, the controller may determine whether the user and the recognized person are related to each other by analyzing the user's SNS (Social Networking Service) usage record using the data stored in the storage unit.

Meanwhile, the control unit may access a database of an external server to determine whether photographing of the recognized object or location is restricted, and if it is determined that photographing is restricted, the image may be corrected.

Alternatively, if the controller recognizes a photographing prohibition sign from the image, the controller may determine a location where the image is photographed as a location where photographing is restricted.

Alternatively, if it is determined that the image needs to be corrected, the control unit performs at least one of mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing on all persons, objects, or images included in the video. can do.

A system for correcting a photographed image according to another embodiment of the present invention for solving the above technical problem includes: a user device that photographs an image, transmits the photographed image to a server, and requests image processing; And at least one of a person, an object, or a location where the image is captured, included in an image received from the user device, and at least one of a relationship between a user of the device and the recognized person, the recognized object, or the recognized location. It may include a server for determining whether or not the image needs to be modified based on one, and if it is necessary to correct the image, transmit the modified image to the user device.

At this time, the server may determine whether or not the recognized person permits the user's photographing by referring to the pre-stored data, and if not, it may be determined that the image needs to be corrected.

In addition, at this time, the server may determine whether or not the photographing of the recognized object or location is restricted by using a database of pre-stored photographing limitation objects or locations, and if it is restricted, it may be determined that image modification is necessary.

Alternatively, the server may receive a time and location at which the image was taken from the user device, and determine whether or not shooting of the recognized person is restricted at the received time and location using previously stored data.

Alternatively, the user device encrypts the captured image using the encryption key received from the server and transmits it to the server, and the server decodes the received image and recognizes at least one of a person, object, or location, and The image can be corrected.

A user device according to another embodiment of the present invention for solving the above technical problem includes: a photographing unit for photographing an image; A communication unit that transmits a photographing notification to a device to be photographed when performing photographing and receives whether photographing is allowed from the device of the photographed person; And a control unit that corrects the captured image according to whether the photographing is allowed.

A method of modifying a photographed image according to another embodiment of the present invention for solving the above technical problem includes: photographing an image on a user's device; Recognizing at least one of a person, an object, or a location where the image is captured in the captured image; And correcting the image based on at least one of a relationship between a user of the device and the recognized person, the recognized object, or the recognized location.

At this time, the modifying step may determine whether the user and the recognized person are related to each other using data stored in the user's device, and if not, modify the image.

In addition, in this case, if the recognized person does not exist in the address book stored in the user device, the modifying step may determine that the user and the recognized person are not related to each other and correct the image.

Alternatively, if the recognized person does not exist in at least one photo previously stored in the user device, the correcting step may determine that the user and the recognized person are not related to each other and correct the image. .

Alternatively, the modifying step may determine whether the user and the recognized person are related to each other by analyzing the user's social networking service (SNS) usage record using data stored in the user device, and the image Can be modified.

A user device according to another embodiment of the present invention for solving the above technical problem includes: a photographing unit for photographing an image; A storage unit in which data is stored; And recognizing a person included in the photographed image, and determining a relationship between a user of the device and the recognized person using data stored in the storage unit, and the user of the device and the recognized person. If not related to each other may include a control unit for modifying the image.

According to an embodiment of the present invention, when an image of a person, object, or location for which shooting is restricted is captured on a user's device, a part or all of the image cannot be identified so as to protect the privacy of the photographed person, and security or It is possible to prevent an individual from obtaining an image of an object or location where shooting is restricted due to a problem of copyright.

1 is a view showing an implementation environment of a method of modifying an image including a shooting restriction element according to an embodiment of the present invention.
2 is a flowchart illustrating a method of modifying an image including a shooting restriction element according to an embodiment of the present invention.
3 to 6 are diagrams for explaining a method of modifying a captured image according to an embodiment of the present invention.
7 to 12 are diagrams illustrating various methods of modifying an image when a person whose shooting is restricted is included in an image according to embodiments of the present invention.
13 to 16 are flowcharts illustrating methods of correcting an image including a person whose shooting is restricted according to embodiments of the present invention.
17 to 18 are diagrams for explaining a method of modifying an image including a shooting restriction element using two or more user devices according to an embodiment of the present invention.
19 is a flowchart illustrating a method of modifying an image including a shooting restriction element using two or more user devices according to an embodiment of the present invention.
20 to 22 are diagrams for explaining a method of modifying an image including a shooting restriction element through communication with a device of a photographed person according to an embodiment of the present invention.
23 and 24 are flowcharts illustrating a method of modifying an image including a photographing restriction element through communication with a device of a photographed subject according to an embodiment of the present invention.
25 to 26 are diagrams for explaining a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention.
28 to 31 are diagrams for explaining a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention.
32 is a flowchart illustrating a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention.
33 to 37 are diagrams for explaining a method of modifying an image including an object to which shooting is restricted according to an embodiment of the present invention.
38 is a flowchart illustrating a method of modifying an image including an object to which shooting is restricted according to an embodiment of the present invention.
39 to 41 are diagrams for explaining a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.
42 is a flowchart illustrating a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.
43 to 45 are diagrams for explaining a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.
46 is a flowchart illustrating a method of correcting an image captured at a location where photography is restricted according to an embodiment of the present invention.
47 to 49 are diagrams for explaining an embodiment of limiting photographing of a person according to privacy protection information according to an embodiment of the present invention.
50 is a flowchart illustrating an embodiment of limiting photographing of a person according to privacy protection information according to an embodiment of the present invention.
51 is a diagram showing the configuration of a user device according to an embodiment of the present invention.
52 is a view for explaining an embodiment of modifying a captured image before storing the captured image in a user terminal in a method of modifying the captured image according to an embodiment of the present invention.
53 is a flowchart for explaining an embodiment of modifying a captured image before storing the captured image in a user terminal in a method of modifying the captured image according to an embodiment of the present invention.
54 and 55 are diagrams for explaining an embodiment of modifying an image stored in a user terminal when transmitting it to the outside in a method of correcting a captured image according to an embodiment of the present invention.
56 is a flowchart for explaining an embodiment of modifying an image stored in a user terminal when transmitting it to the outside in a method of modifying a captured image according to an embodiment of the present invention.
FIG. 57 is a view for explaining an embodiment in which a method of correcting a captured image according to an embodiment of the present invention transmits the captured image to the outside without directly storing the captured image in a user terminal.
58 is a flowchart for explaining an embodiment in which a captured image is directly modified and transmitted to the outside without storing the captured image in the user terminal in the method of correcting the captured image according to an embodiment of the present invention.
59 to 61 are diagrams for explaining an embodiment of modifying an image photographed at a location where photographing is restricted according to an embodiment of the present invention.
62 is a flowchart illustrating a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.
63 to 66 are diagrams for explaining an embodiment of modifying a photographed image according to a user's selection according to an embodiment of the present invention.
67 and 68 are flowcharts illustrating a method of correcting a photographed image according to a user's selection according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In order to more clearly describe the features of the present embodiments, detailed descriptions of matters well known to those skilled in the art to which the following embodiments pertain will be omitted.

1 is a view showing an implementation environment of a method of modifying an image including a shooting restriction element according to an embodiment of the present invention.

Referring to FIG. 1, the user A 1 may take an image including a person or an object, or an image for a specific location using the smart glass 100. In this case, the captured image includes not only a still image such as a photograph, but also a moving image.

As illustrated, the smart glass 100 may photograph an image including a person (user B (2)), an object (sculpture 20), or a photograph prohibition sign 30. The smart glass 100 recognizes at least one of a person, an object, or a location in which the image is captured in the captured image, and an element in which image is restricted based on the recognition result, that is, a photography restricted element It is possible to determine whether is included.

Here, the photographing restricted element means a person, object, or location where photography is restricted due to privacy, copyright, or security reasons. The photographing limiting element may be applied to all persons at once, or the photographing limiting element may vary depending on the photographer. For example, photographing a military security area is restricted to everyone, but photographing a specific person may be restricted to a person who does not know the specific person. If a person or object for which shooting is restricted is included in an image, or if an image for a location where shooting is restricted is included, it is determined that the image includes a shooting restriction element.

Meanwhile, when the smart glass 100 photographs an image and transmits the photographed image to another device or server connected to the smart glass 100, the device or server receiving the image performs recognition of a person, object, or location, and Also, it may be determined whether the image includes a shooting restriction element.

When it is determined that the image includes the shooting limiting element, the smart glasses 100 and other devices or servers connected to the smart glasses 100 modify the captured images. In detail, the image is modified so that it is not possible to identify the shooting restriction element from the captured image. Detailed methods of modifying the image will be described in detail with reference to the drawings below.

On the other hand, the user A (1) can take pictures using various devices in addition to the smart glass 100, such as a wearable device, a smart phone 100-2, or a tablet 100-5, such as the smart watch 100-1. A mobile device such as), a camera 100-3, or a general imaging device such as a camcorder 100-4 may be used. In addition, it is obvious that various devices capable of taking a picture or a video can be applied to an embodiment of the present invention even if the device is not mentioned above.

In general, the device that is carried by the most users is the smartphone 100-2, and it is highly likely that the user's information necessary for the implementation of the present invention is stored in the smartphone 100-2, and the smartphone 100-2 ) Is also possible to perform processing such as image modification. In the following embodiments, it is assumed that the smartphone 100-2 is used as a user device.

2 is a flowchart illustrating steps of a method of modifying an image including a shooting restriction element according to an embodiment of the present invention.

Referring to FIG. 2, an image is captured on the user's device in step 201. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 202, the user's device recognizes at least one of a person, an object, or a location where the image is captured in the captured image. In this case, as a specific method in which the user's device recognizes a person included in an image, a face recognition method may be used, for example. However, the present invention is not limited thereto, and various methods of identifying a person may be used. Face recognition specifically includes processes for detecting a face from an image and extracting features from the detected face, and techniques related to face recognition are widely known, so detailed descriptions thereof are omitted herein.

On the other hand, the user's device may perform recognition of the object by detecting an object included in the image and extracting features such as the shape and color of the detected object.

On the other hand, the method of recognizing the location where the image is captured by the user's device includes a method of recognizing an absolute location, a method of determining whether it is in a specific place, or a method of determining whether it is a location where the shooting is restricted. Can be.

First, the method of recognizing the absolute position may be performed by a method in which the user's device performing the shooting recognizes an absolute position such as latitude and longitude at the time when the shooting is performed through a built-in GPS (Global Positioning System). have.

Second, the method of determining whether or not a particular place is applicable may be performed in a manner of determining whether a device of a user performing shooting is located in a specific place (eg, a museum, etc.) at the time when the shooting is performed. have. For example, when a user's device is located in a specific place, it may be determined that the user's device is located in a specific place through communication with a server provided in the place. Alternatively, the user's device may recognize the absolute location using GPS and compare it with the absolute location of a specific location previously stored in the database.

Third, the method of determining whether the photographing is limited to a location where the photographing is restricted may be performed in a manner of determining whether the device of the user who performs photographing is located in the photographing prohibition zone at the time when photographing is performed. For example, a user's device may recognize and analyze a sign (e.g. a photographing prohibition zone sign) included in the photographed image to determine whether or not the photographing prohibition zone. Alternatively, the user's device may recognize the absolute location using GPS and then compare it with the absolute location of the prohibition zone previously stored in the database.

If recognition of at least one of a person, an object, or a location in which the image is captured in the image is completed, in step 203, the user's device relates to the user of the device and the recognized person, the recognized object, or the recognized location. Based on at least one of the above, it is determined whether or not the photographing restriction element is included in the image. That is, the user's device determines whether the image needs to be corrected. If an image capture restriction element is included in the image, the user's device determines that the image needs to be corrected, and if not, it is determined that no correction is necessary.

At this time, the relationship between the user of the device and the recognized person, that is, the relationship between the photographer and the subject, whether the photographer and the subject know each other, whether the subject allowed the photographer to take his or her photograph, or avoided It means whether the photographer is a celebrity, so whether or not the photographer's shooting is not restricted. Based on the relationship between the user of the device and the recognized person, if the photographing of the photographed person included in the image is restricted, it is determined that the photographing restriction element is included in the image.

On the other hand, if the recognized object is an object to which shooting is restricted, the user's device determines that the image has a shooting restriction element. Objects for which photography is restricted may be objects that are subject to copyright protection, for example, artworks, or objects for which photography of the general public is restricted for various reasons.

The detailed determination of whether the recognized object corresponds to an object to which shooting is restricted may be performed in various ways. For example, a database for an object whose shooting is restricted is stored in a memory in the user's device, and the user's device Recognized objects can be compared to stored databases. Alternatively, a database for an object to which shooting is restricted is stored in an external server, and the user's device may communicate with the server to access the database and compare the recognized object with the database. Alternatively, when the user's device enters a specific place, such as a museum, the server managing the corresponding place may receive a database of objects whose shooting is restricted in the corresponding place, and compare it with the recognized object. In addition, the determination of whether the recognized object is an object to which shooting is restricted may be performed in various ways.

On the other hand, if the recognized location is a location where photography is restricted, the user's device determines that the image capture restriction element is included in the image. The location where photography is restricted may be a location where photography is restricted for security reasons, such as a military security area, or may be a location where photography is restricted for copyright reasons such as a venue. In addition, there may be locations where photography of the general public is restricted for various reasons.

The detailed determination of whether the recognized location corresponds to a location where shooting is restricted can be performed in various ways. For example, a database of locations where shooting is restricted is stored in a memory in the user's device, and the user's device The recognized location can be compared with the stored database. Alternatively, a database on a location where shooting is restricted is stored in an external server, and the user's device can communicate with the server to access the database and compare the recognized location with the database. Alternatively, the user's device may recognize and analyze a sign included in the captured image (e.g., a photographing prohibition zone) and determine whether or not the photographing prohibition zone.

If the image capturing restriction element is included in the image in operation 204, the process proceeds to operation 205 where the user's device modifies the image. In detail, the user's device may modify the image so that the shooting restriction element included in the image cannot be identified. For example, the user's device may mosaic or blur a person or object for which photographing is restricted. Alternatively, the user's device may be synthesized with a screen in a captured image excluding a person or object whose shooting is restricted, or a preset alternative image may be synthesized. On the other hand, if the image is photographed at a location where the photographing is restricted, the user's device may perform mosaic processing, blur processing, or synthesis of an alternate image preset in the entire image so that the entire image cannot be identified. In addition, the user's device may modify the image so that it is impossible to identify the shooting restriction element included in the image in various ways. The specific method of modifying the image will be described in detail with reference to the corresponding drawings below.

In step 204, if the photographing restriction element is not included in the image, the process ends without modifying the image.

On the other hand, in the above, steps 202 to 205 of FIG. 2 are described as being performed on the device of the user who performed the shooting. Alternatively, some or all of the above steps may be performed on another device or server connected to the user's device. have. Detailed examples will be described below.

3 to 6 are diagrams for explaining a method of modifying an image including a shooting restriction element according to an embodiment of the present invention.

Referring to FIG. 3, an image including the user B (2) and the user C (3) is photographed on the smartphone 100-2 of the user A (1). Subsequently, the smartphone 100-2 of the user A 1 determines whether a person whose shooting is restricted is included in the captured image and corrects the image when the person whose shooting is restricted is included.

4 to 6 show examples of determining whether the smartphone 100-2 of the user A 1 includes a person whose shooting is restricted in the image by different methods, and correcting the image, respectively.

Referring to FIG. 4, the smartphone 100-2 recognizes the faces of the user B (2) and the user C (3) by performing face recognition on the captured original image 400 a. The smartphone 100-2 compares faces recognized in the image with pictures included in the address book stored in the smartphone 100-2, respectively. The address book stored in the smartphone 100-2 may store a picture of a person who corresponds with the name and phone number.

In FIG. 4, the smartphone 100-2 compares the faces recognized in the original image 400a with the pictures included in the address book, and confirms that the picture of the user B2 exists in the address book. The smartphone 100-2 determines that the user B 2 having a picture in the address book is allowed to be photographed.

However, the smartphone 100-2 determines that the user C (3) is a person whose shooting is restricted if a picture matching the user C (3) cannot be found in the address book. In other words, the smartphone 100-2 may determine that the user A (1) and the user C (3) are not related to each other. Therefore, the smartphone 100-2 determines that the photographing restriction element is included in the image and corrects the image. For example, the smartphone 100-2 may modify a part of the image corresponding to the user C (3) so that the recognized user C (3) cannot be identified. Specifically, as shown in FIG. 4, the smartphone 100-2 performs mosaic processing on the face region of the user C 3, which is a person whose shooting is restricted, and outputs a corrected image 400b.

The reason for determining whether the photographing restriction element is included by comparing the face recognized by the smartphone 100-2 with the picture included in the address book is the address book of the smartphone 100-2 of the user A (1). This is because the registered person is likely to have a relationship with the user A (1). That is, the smartphone 100-2 uses the stored address book to grasp the relationship between the user A (1) and the subjects, and based on this, determines whether or not the shooting restriction element is included.

Referring to FIG. 5, the smartphone 100-2 recognizes the faces of the user B (2) and the user C (3) by performing face recognition on the photographed original image 500a. The smartphone 100-2 compares faces recognized in the image with pictures stored in the album of the smartphone 100-2, respectively.

In FIG. 5, the smartphone 100-2 compares the faces recognized in the original image 500a with the pictures stored in the album, and confirms that the user B 2 is included in at least one of the pictures stored in the album. . The smartphone 100-2 determines that the user B 2 included in the picture stored in the album is a person allowed to shoot.

However, the smartphone 100-2 determines that the user C (3) is a person whose shooting is restricted if the photo containing the user C (3) cannot be found in the album. In other words, the smartphone 100-2 may determine that the user A (1) and the user C (3) are not related to each other. Therefore, the smartphone 100-2 determines that the photographing restriction element is included in the image and corrects the image. For example, the smartphone 100-2 may modify a part of the image corresponding to the user C (3) so that the recognized user C (3) cannot be identified. Specifically, as shown in FIG. 5, the smartphone 100-2 performs mosaic processing on the face region of the user C (3), which is a person whose shooting is restricted, and outputs the corrected image 500b.

The reason for determining whether the photographing restriction element is included by comparing the face recognized by the smartphone 100-2 with the pictures stored in the album is the album of the smartphone 100-2 of the user A (1). This is because the person included in the stored picture is more likely to have a relationship with the user A 1. That is, the smartphone 100-2 uses the stored album's photos to grasp the relationship between the user A (1) and the subjects, and based on this, determines whether the photographing restriction element is included.

Referring to FIG. 6, the smartphone 100-2 recognizes the faces of the user B (2) and the user C (3) by performing face recognition on the captured original image 600a. The smartphone 100-2 analyzes the social networking service (SNS) usage record of the user A (1) to grasp the relationship between the recognized persons and the user A (1).

In FIG. 6, as a result of analyzing the usage record of the SNS, the smartphone 100-2 determines that the user B (2) knows each other with the user A (1), and the person allowed to photograph the user B (2) Judging by. For example, if user A (1) and user B (2) are registered as friends on SNS, or if one of them has commented on the post registered on SNS, user A (1) and user B (2) can be judged to know each other.

However, as a result of analyzing usage records of the SNS of the smartphone 100-2, if it is determined that the user C (3) is not in contact with the user A (1), the user C (3) is restricted from shooting Judging by. In other words, the smartphone 100-2 may determine that the user A (1) and the user C (3) are not related to each other. Therefore, the smartphone 100-2 determines that the photographing restriction element is included in the image and corrects the image. For example, the smartphone 100-2 may modify a part of the image corresponding to the user C (3) so that the recognized user C (3) cannot be identified. Specifically, as shown in FIG. 6, the smartphone 100-2 performs mosaic processing on the face area of the user C 3, which is a person whose shooting is restricted, and outputs a corrected image 600b.

7 to 12 illustrate various methods of modifying an image when a shooting restriction element is included in an image according to embodiments of the present invention.

7 shows an example of performing a blur treatment on a photographing limiting element. Referring to FIG. 7, the original image 700a includes a user B (2) and a user C (3). It is assumed that user C (3) falls under the shooting limit element. Therefore, the area where the user C (3) is displayed in the image is blurred. Referring to the modified image 700b, it can be confirmed that the user C (3) has been processed in a blur. Meanwhile, since a specific method of performing blur processing on a specific area of an image is a generally well-known technique, a detailed description is omitted here.

FIG. 8 shows an example of combining the shooting restriction element with the background screen. Referring to FIG. 8, a user B (2) and a user C (3) are included in the original image 800a. It is assumed that user C (3) falls under the shooting limit element. Therefore, the user C (3) in the image is synthesized with the background screen.

At this time, the method for synthesizing the user C (3) with the background screen, specifically deleting the user C (3) from the image, and deleting the user C (3) with an image similar to the background screen around the user C (3) Fill the area.

Referring to the modified image 800b of FIG. 8, the edge 3-1 of the user C (3) is the same as or similar to the pixels existing outside the edge 3-1 of the border 3-1. Fill the inside. For example, the inner pixel 802 adjacent to the border 3-1 is set to have the same or similar color, saturation, and brightness as the outer pixel 801 adjacent to the border 3-1. In addition, among the pixels inside the border 3-1, other pixels adjacent to the pixel 802 are set to have similar color, saturation and brightness as the pixel 802. Meanwhile, various methods commonly used in the art may be applied to a method of synthesizing a specific region of an image with a background screen, so a detailed description thereof will be omitted here.

FIG. 9 shows an example of synthesizing a background image previously stored in an image including a shooting restriction element. Referring to FIG. 9, the original image 900a includes a user B (2), a user C (3), and a structure 901. It is assumed that user C (3) falls under the shooting limit element. Referring to the original image 900a, the user C 3 is displayed to partially overlap the structure 901. Accordingly, the user C (3) is deleted from the image, and the area where the user C (3) is deleted is filled with an image of the structure 901 stored in advance. Referring to the modified image 900b, it can be seen that a portion of the region where the user C 3 is displayed in the image overlapping the structure 901 has been replaced with an image of the structure 901 previously stored. At this time, the entire image of the structure 901 may be stored in advance in a user device or a server that corrects an image.

FIG. 10 shows an example of synthesizing a photographing restriction element with a pre-stored alternative image. Referring to FIG. 10, a user B (2) and a user C (3) are included in the original image 1000a. It is assumed that user C (3) falls under the shooting limit element. The modified image 1000b is generated by replacing the user C 3 with the pre-stored replacement image 1001. In FIG. 10, the replacement image 1001 was selected as an image to replace the face of the person. However, alternatively, an image may be selected to replace the entire person.

Meanwhile, in synthesizing the substitute image on the face of the person corresponding to the photographing restriction element, the alternative image may be synthesized according to the face direction and angle of the person in the original image to obtain a more natural image. Examples of this are shown in FIGS. 11 and 12.

Referring to FIG. 11, the face of a person included in the original image 1100a faces the front. Therefore, as shown in the corrected image 1100b, an alternative image 1110 facing the front according to the face direction and angle of the person in the original image 1100a was synthesized in the face portion of the person.

On the other hand, referring to Figure 12, the face of the person included in the original image 1200a is looking at the side. Therefore, as shown in the modified image 1200b, an alternative image 1210 looking at the side according to the face direction and angle of the person in the original image 1200a is synthesized in the face part of the person.

As described above, a more natural image can be obtained by synthesizing a corresponding alternative image according to the face direction and angle of the person.

13 to 16 are flowcharts illustrating methods of correcting an image including a person whose shooting is restricted according to embodiments of the present invention.

Referring to FIG. 13, in step 1301, an image including a person is photographed on a user device. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 1302, the user device performs face recognition of the person included in the captured image. Specifically, face recognition may be performed by detecting a face region in an image and extracting features from the detected face region.

In step 1303, the user device determines the degree of association between the photographer and the subject using the data stored in the user device. That is, the user device determines whether or not the photographer and the subject are aware of each other.

In step 1304, the user device determines whether a person not related to the photographer is included in the captured image. At this time, a specific method of determining the degree of association between the photographer and the subject and determining whether the image includes a person who is not related to the photographer may be implemented in various ways. 16.

As a result of the determination, if a person not related to the photographer is included in the captured image, the process proceeds to step 1305 and the user device performs a protection process for the person not related to the photographer. That is, the user device modifies the image so that a person not related to the photographer cannot be identified from the image. The specific method of modifying the image is as described above with reference to FIGS. 7 to 12.

Referring to FIG. 14, in step 1401, an image including a person is photographed on a smartphone. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 1402, the smartphone performs face recognition of the person included in the captured image. Specifically, face recognition may be performed by detecting a face region in an image and extracting features from the detected face region.

In step 1403, the smartphone determines whether the recognized person exists in the address book of the smartphone. In detail, it is determined whether the recognized person matches at least one photo included in the address book of the smartphone. As a result of the determination, if the recognized person does not exist in the address book of the smartphone, the process proceeds to step 1404.

In step 1404, the smartphone performs mosaic processing on the recognized person. Meanwhile, the smartphone may perform blur processing, synthesis processing with a background screen, and the like on the recognized person.

Referring to FIG. 15, in step 1501, an image including a person is photographed on a smartphone. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 1502, the smartphone performs face recognition of the person included in the captured image. Specifically, face recognition may be performed by detecting a face region in an image and extracting features from the detected face region.

In step 1503, the smartphone determines whether the recognized person exists in the photo stored in the album of the smartphone. In detail, it is determined whether or not a person matching the recognized person is included in at least one picture stored in the album. As a result of the determination, if the recognized person does not exist in the picture stored in the album of the smartphone, the process proceeds to step 1504.

In step 1504, the smartphone performs mosaic processing on the recognized person. Meanwhile, the smartphone may perform blur processing, synthesis processing with a background screen, and the like on the recognized person.

Referring to FIG. 16, in step 1601, an image including a person is photographed on a smartphone. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 1602, the smartphone performs face recognition of the person included in the captured image. Specifically, face recognition may be performed by detecting a face region in an image and extracting features from the detected face region.

In step 1603, the smartphone analyzes the SNS usage record of the photographer, that is, the smartphone user. Various records of the user's use of the smartphone may be stored in the smartphone, and among them, SNS usage records are analyzed.

In step 1604, the smartphone determines whether the recognized person is registered as an acquaintance of the photographer on the SNS. As a result of the determination, if the recognized person is not registered as an acquaintance of the photographer on the SNS, the process proceeds to step 1605.

In step 1605, the smartphone performs mosaic processing on the recognized person. Meanwhile, the smartphone may perform blur processing, synthesis processing with a background screen, and the like on the recognized person.

Meanwhile, in FIG. 14 to FIG. 16, the case where the user device is a smart phone is exemplified, but differently, various devices including an address book and a picture can be used, such as a tablet.

17 to 18 are diagrams for explaining a method of modifying an image including a shooting restriction element using two or more user devices according to an embodiment of the present invention. In particular, in the embodiments shown in Figs. 17 to 19, the user device performing the imaging and the user device performing the image processing are separated from each other.

Referring to FIG. 17, the user A 1 photographs an image including a person using his wearable device, smart glasses 100. At this time, the captured image includes user B (2) and user C (3). The smart glass 100 transmits the photographed image to the smartphone 100-2, which is also the device of the user A (1). The smartphone 100-2 that has received the captured image determines whether a captured restriction element is included in the received image, and if it is judged to be included, corrects the image.

Referring to FIG. 18, the smartphone 100-2 performs face recognition of people included in the original image 1800a received from the smart glass 100. When the face recognition of the user B (2) and the user C (3) included in the original image 1800a is completed, the smartphone 100-2 stores the recognized faces of each person on the smartphone 100-2. Compare the photos in the address book. The address book stored in the smartphone 100-2 may store a picture of a person who corresponds with the name and phone number.

In FIG. 18, when the smartphone 100-2 compares the faces recognized in the original image 1800a with the pictures included in the address book, if it is confirmed that the picture of the user B (2) exists in the address book, the smartphone (100-2) judges the user B (2) as a person who is allowed to photograph. However, the smartphone 100-2 determines that the user C (3) is a person whose shooting is restricted if a picture matching the user C (3) cannot be found in the address book. Therefore, the smartphone 100-2 determines that the photographing restriction element is included in the image and corrects the image. For example, the smartphone 100-2 may output a modified image 1800b by combining the user C 3, which is a person whose shooting is restricted, with a background screen. A detailed method of synthesizing user C (3) with a background screen is as described above with reference to FIG. 8.

In addition, the smartphone 100-2 compares faces recognized in the original image 1800a with pictures stored in the album of the smartphone 100-2, or analyzes SNS usage records of the user A1. You can also judge the person whose shooting is restricted. For detailed methods, refer to the parts described with reference to FIGS. 5 and 6 above.

In addition, although FIG. 18 shows an example in which the smartphone 100-2 synthesizes the user C (3) with a background screen, in addition, the user C (3) is mosaic-processed, blur-processed, or synthesized as an alternative image. The image can be modified in various ways. For details, refer to FIGS. 7 to 12.

19 is a flowchart illustrating a method of modifying an image including a shooting restriction element according to an embodiment of the present invention.

Referring to FIG. 19, in step 1901, an image including a person is photographed in a wearable device. In this case, the wearable device may be a wearable device equipped with a photographing function, such as a smart glass or a smart watch.

In step 1902, the wearable device transmits the captured image to the smartphone. At this time, the image is transmitted to the smartphone owned by the user of the wearable device, and a device such as a tablet may be used instead of a smartphone.

In step 1903, the smartphone receiving the image performs face recognition of the person included in the image. Specifically, face recognition may be performed by detecting a face region in an image and extracting features from the detected face region.

In step 1904, the smartphone determines whether the recognized person exists in the address book of the smartphone. That is, the smartphone compares the recognized face with the pictures stored in the address book to determine whether there is a matching picture.

As a result of the determination, if the recognized person does not exist in the address book of the smartphone, the process proceeds to step 1905 to synthesize the recognized person with the background screen.

20 to 22 are diagrams for explaining a method of modifying an image including a shooting restriction element through communication with a device of a photographed person according to an embodiment of the present invention. In particular, in the embodiment shown in FIGS. 20 to 22, the device of the person to be photographed transmits whether or not photography is permitted to the device of the photographer.

Referring to FIG. 20, an image including the user B 2 is photographed on the smartphone 100-2 of the user A 1. When photographing, the smartphone 100-2 of the user A (1) sends a photographing notification to the smartphone 200 of the user B (2), which is the subject. At this time, the transmission notification to be transmitted may include identification information that can identify the user A (1).

The smartphone 200 of the user B (2) receiving the shooting notification confirms that the photographer is the user A (1) by using the identification information included in the shooting notification, and determines whether to allow the shooting of the user A (1). Judge. A detailed method of identifying the photographer using the identification information included in the shooting notification and determining whether to allow the shooting will be described below with reference to FIG. 21.

When determining whether the photographing is allowed, the smartphone 200 of the user B 2 transmits whether photographing is allowed to the smartphone 100-2 of the user A 1. The smartphone 100-2 of the user A (1) stores the image with or without modification depending on whether the received shooting is allowed.

In FIG. 21, the smartphone 100-2 of the user A (1) sends a shooting notification to the smartphone 200 of the user B (2), and the smartphone 200 of the user B (2) allows shooting After the determination, the process of transmitting it to the smartphone 100-2 of the user A (1) is illustrated.

Referring to FIG. 21, the smartphone 100-2 of the user A (1) includes identification information of the user A (1) as the smartphone 200 of the user B (2) when photographing the user B (2). To send a shooting notification. At this time, the identification information included in the shooting notification may include the user A (1) 's name and photo.

The user B (2) 's smartphone 200 uses the identification information of the user A (1) included in the shooting notification received from the user A (1)' s smartphone 100-2 to determine whether or not shooting is allowed. . In detail, the smartphone 200 of the user B (2) compares the picture of the user A (1) included in the identification information with the pictures stored in the address book, or the name of the user A (1) included in the identification information Is compared to the names stored in the address book. As a result of comparison, if the picture or name of the user A (1) does not exist in the address book, the smartphone 200 determines that shooting is not permitted. Conversely, if a picture or a name of the user A (1) exists in the address book, the smartphone 200 determines that shooting is permitted.

On the other hand, the smartphone 200 of the user B (2) compares the photo of the user A (1) included in the identification information with the photo stored in the album of the smartphone 200, or uses the SNS of the user B (2) It is also possible to determine whether or not shooting is allowed through analysis of records.

In FIG. 21, the smartphone 200 of the user B (2) determines that shooting is impossible if there is no picture of the user A (1) in the address book, and the smartphone 200-2 of the user A (1) Send a message that cannot be taken. At this time, the smartphone 200 of the user B (2) transmits a picture of the user B (2) to the smartphone (100-2) of the user A (1) with a message that can not be taken.

Subsequently, referring to FIG. 22, the smartphone 100-2 of the user A 1 performs face recognition on the captured original image 1600a. When the face recognition is completed, the smartphone 100-2 of the user A (1) receives the photograph and the original image of the user B (2) along with a non-shootable message from the smartphone 200 of the user B (2) ( 1600a). As a result of comparison, if it is determined that the recognized face matches the picture of the user B (2) received with the non-shootable message, the smartphone 100-2 of the user A (1) displays the user B () in the original image 1600a. The face portion of 2) is mosaic-processed to output a modified image 1600b. Meanwhile, in addition to the mosaic processing, the smartphone 100-2 of the user A (1) may modify the image by a method such as blur processing, synthesis with a background screen, or synthesis with an alternative image.

23 is a flowchart illustrating a method of modifying an image including a photographing restriction element through communication with a device to be photographed according to an embodiment of the present invention.

Referring to FIG. 23, in step 2301, the photographer's device transmits a shooting notification including the photographer's identification information to the device of the photographer. For example, the device of the photographer may search for a device located within a certain distance around when performing the shooting, and transmit a shooting notification to the found device. Short-range wireless communication such as Wi-Fi Direct or Bluetooth may be used to search for a device and transmit a shooting notification, and consent may be required on the device of the subject to receive the shooting notification.

In step 2302, the device of the photographed person determines whether photographing is allowed using the identification information of the photographer included in the received photographing notification.

As a result of the determination, if it is determined that shooting is allowed, the process proceeds to step 2307 through step 2303. In step 2307, the device to be photographed transmits a shooting permission message to the device of the photographer. The device of the photographer who has received this determines that the shooting restriction element is not included in the captured image, and stores the image without modification.

Meanwhile, if it is determined that shooting is not allowed as a result of the determination, the process proceeds to step 2303 and step 2304. In step 2304, the device of the to-be-photographed transmits a non-shooting message to the photographer's device along with the picture of the to-be-photographed.

In step 2305, the device of the photographer performs face recognition of the people included in the captured image.

In step 2306, the device of the photographer compares the faces recognized in the captured image with the photo of the subject to be received, and protects the face matching the photo of the subject. That is, mosaic processing, blur processing, background screen and synthesis processing, or alternative image and synthesis processing are performed on the face matching the photographed subject.

FIG. 24 is a flowchart of a specific embodiment of determining whether or not photographing is allowed using the address book of the device to be photographed.

Referring to FIG. 24, in step 2401, the smartphone of the photographer transmits a shooting notification including identification information of the photographer to the smartphone of the photographer.

In step 2402, the subject's smartphone determines whether the photographer exists in the address book using the received identification information of the photographer. In detail, the name or photo of the photographer included in the identification information of the photographer is compared with the names or photos stored in the address book. On the other hand, the smartphone of the photographed person may determine whether or not the photographing is allowed by comparing the identification information of the photographed person with an album stored in the photographed smartphone or by analyzing the record of the SNS use of the photographed person.

As a result of the determination, if a photographer exists in the address book, it is determined that shooting is allowed, and the process proceeds to step 2406, where the smartphone of the subject sends a shooting permission message to the photographer's smartphone.

On the other hand, as a result of the determination, if there is no photographer in the address book, it is determined that shooting is not permitted and the process proceeds to step 2403.

In step 2403, the subject's smartphone transmits a non-shooting message to the photographer's smartphone along with the photograph of the subject.

In step 2404, the smartphone of the photographer performs face recognition of the people included in the captured image.

In step 2405, the smartphone of the photographer compares the faces recognized in the photographed image with the photographed subject, and protects the face matching the photographed subject. That is, mosaic processing, blur processing, background screen and synthesis processing, or alternative image and synthesis processing are performed on the face matching the photographed subject.

25 to 26 are diagrams for explaining a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention.

Referring to FIG. 25, the smartphone 100-2 of the user A (1) photographs an image including the user B (2) and the user C (3), and transmits the photographed original image to the server 300 And request image processing. The server 300 determines whether the photographing restriction element is included in the received original image, and if it is determined that it is included, modifies the image and transmits the image to the smartphone 100-2 of the user A (1). A detailed method for determining whether the server 300 includes a shooting restriction element will be described in detail with reference to FIG. 26 below.

Referring to FIG. 26, the server 300 performs face recognition of the user B (2) and the user C (3) on the original image 2600a received from the smartphone 100-2 of the user A (1). . The server 300 determines whether the recognized persons are allowed to shoot by referring to the pre-stored privacy protection information 2610.

At this time, the privacy protection information 2610 defines another user who is allowed to shoot for at least one user. Referring to FIG. 26, other users who are allowed to shoot for each user are mapped to the privacy protection information 2610. For example, user A does not allow other users to shoot at all. In addition, user B only allows user A to be photographed, and user C allows user B and user D to be photographed.

The server 300 compares the faces of the user B (2) and the user C (3) recognized in the original image 2600a with the privacy protection information 2610 to determine whether or not shooting is allowed. According to the privacy protection information 2610, the user B (2) allows the user A (1) to be photographed, and the user C (3) does not allow the user A (1) to be photographed, so the server 300 is a user C (3) is judged as a shooting limit factor.

On the other hand, in order to determine whether shooting is allowed, the server 300 needs to confirm that the photographer of the original image 2600a is the user A (1). To this end, the smartphone 100-2 of the user A (1) is the original image The identification information of the user A 1 may be transmitted to the server 300 together with the 2600a.

When the server 300 confirms that the user C (3) is a shooting restriction element, the original image 2600a mosaics the user C (3) to generate a corrected image 2600b, and the generated corrected image 2600b is used by the user. It transmits to the smartphone 100-2 of A (1). Meanwhile, the server 300 may modify an image through blur processing, background screen synthesis processing, or alternative image synthesis processing in addition to mosaic processing.

27 is a flowchart illustrating a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention.

Referring to FIG. 27, in step 2701, an image is captured on the user device.

In step 2702, the user device transmits the captured image to the server and requests image processing.

In step 2703, the server performs face recognition of the person included in the captured image received from the user device.

In step 2704, the server determines whether to allow the recognized person to shoot with reference to the pre-stored privacy protection information. That is, the server determines whether photographing of the recognized person is permitted to the photographer of the received photographed image. 26 for a detailed method of determining whether to allow shooting using the privacy protection information.

If it is determined in step 2705 that the recognized person does not allow the user to shoot, the server proceeds to step 2706 to perform a protection process for the person in the received captured image. Subsequently, in step 2707, the modified image is transmitted to the user device.

Meanwhile, if it is determined in step 2705 that the recognized person allows the user to shoot, the server proceeds to step 2707 and transmits the received image to the user device without modification.

28 to 31 are diagrams for explaining a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention. In particular, in the embodiment shown in FIGS. 28 to 31, the user device performing the shooting encrypts the captured image using the encryption key managed by the server and then transmits the captured image to the server, so that the photographer is the original that has not been processed by the server Video is not accessible.

Referring to FIG. 28, the smartphone 100-2 of the user A 1 receives an encryption key from the server 300. At this time, the encryption key is managed by the server 300. The smartphone 100-2 of the user A (1) photographs the user B (2) and the user C (3), and encrypts the captured image using the encryption key received from the server 300. At this time, since the smartphone 100-2 of the user A (1) immediately encrypts the image after shooting, the user A (1) cannot access the original image.

As shown in FIG. 29, the smartphone 100-2 of the user A 1 transmits the encrypted image to the server 300.

Referring to FIG. 30, the server 300 decodes the received image. Since the encryption key used to encrypt the image is managed by the server 300, the server 300 can decrypt the received image using this. The server 300 recognizes the faces of the user B (2) and the user C (3) by performing face recognition on the decoded image. After performing the face recognition, the server 300 determines whether or not the recognized persons are allowed to shoot using the pre-stored privacy protection information 3010. A specific method of determining whether to allow shooting using the privacy protection information 3010 is as described above with reference to FIG. 26.

The server 300 determines whether or not shooting is permitted with reference to the privacy protection information 3010, and if it is determined that the shooting of the user C 3 is not allowed, the user C 3 is mosaic-processed in the image to correct the image.

Subsequently, the server 300 transmits the modified image to the smartphone 100-2 of the user A 1 as shown in FIG. 31.

32 is a flowchart illustrating a method of modifying an image including a shooting restriction element using a server according to an embodiment of the present invention.

Referring to FIG. 32, in step 3201, the user device photographs an image including a person.

In step 3202, the user device encrypts the captured image using the encryption key received from the server. At this time, the encryption key is managed by the server.

In step 3203, the user device transmits the encrypted image to the server.

In step 3204, the server decodes the image received from the user device and performs face recognition of the people included in the image.

In step 3205, the server determines whether to allow the recognition of the recognized person by referring to the pre-stored privacy protection information. A specific method of determining whether shooting is permitted with reference to the privacy protection information is as described above with reference to FIG. 26.

In step 3206, the server performs a protection process for the person who is not allowed to shoot. For example, mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing are performed on persons who are not allowed to shoot among the persons included in the image.

In step 3207, the server transmits the modified image to the user device.

33 to 37 are diagrams for explaining a method of modifying an image including an object to which shooting is restricted according to an embodiment of the present invention.

Referring to FIG. 33, the smartphone 100-2 of the user A 1 photographs an image including the user B 2 and the sculpture 20. The smartphone 100-2 of the user A (1) transmits the captured image to the server 300 and requests image processing. The server 300 determines whether or not the photographing restriction element is included in the received image, and if it is determined, corrects the image and transmits the corrected image to the smartphone 100-2 of the user A (1). . At this time, the server 300 determines whether or not the image capture limiting element is included, and examples of modifying the image are illustrated in FIGS. 34 to 37.

Referring to FIG. 34, the server 300 recognizes an object included in the original image 3400a. That is, the sculpture 20 is recognized by extracting features such as shape and color of the sculpture 20 included in the original image 3400a. The server 300 determines whether or not the recognized sculpture 20 exists in the photographing restriction object database stored in the server 300. In the photography-restricted objects database, photographs of objects for which photography is restricted are stored.

As a result of the determination, if the recognized sculpture 20 is included in the object database for shooting restriction, the server 300 determines the sculpture 20 as a shooting restriction element. The server 300 mosaicizes the sculpture 20 from the original image 3400a to generate a modified image 3400b, and uses the generated modified image 3400b as the smartphone 100-2 of the user A (1). send.

Meanwhile, in FIGS. 35 to 37, embodiments for modifying an image in different ways are illustrated. The method in which the server 300 recognizes an object in an image in FIGS. 35 to 37 and determines whether an image capturing restriction element is included in an image using a pre-stored shooting restriction object database is as described in FIG. 34. However, in modifying the image, in FIG. 35, a blur process is performed to generate a corrected image 3500b. In FIG. 36, a modified image 3600b is generated by performing a background screen synthesis process, and a detailed method of combining the background screen is as described in FIG. 8 above. In FIG. 37, a modified image 3700b is generated by synthesizing the replacement image 3710.

38 is a flowchart illustrating a method of modifying an image including an object to which shooting is restricted according to an embodiment of the present invention.

Referring to FIG. 38, in step 3801, an image is captured on the user's device.

In step 3802, the user's device transmits the captured image to the server.

In step 3803, the server performs recognition of objects included in the captured image. Specifically, an object included in an image is detected, and features such as the shape and color of the detected object are extracted.

In step 3804, the server determines whether the recognized object exists in the pre-stored object database. In the photography-restricted objects database, a list of objects for which photography is restricted is stored.

As a result of the determination, if the recognized object is present in the object database for shooting restrictions, the process proceeds to step 3805, the server performs a protection process on the recognized object to correct the image, and in step 3806, the server transmits the modified image to the user device. do.

However, as a result of the determination, if the recognized object does not exist in the shooting restriction object database, the process proceeds to step 3806, and the server transmits the image to the user device without modifying the image.

39 to 41 are diagrams for explaining a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.

Referring to FIG. 39, the smartphone 100-2 of the user A (1) takes a picture at a venue where shooting is restricted, sends the captured image to the server 300, and requests image processing. At this time, the smartphone 100-2 of the user A (1) transmits location information to the server 300 together with the captured image. The transmitted location information is information about the location where the smartphone 100-2 has taken the photograph, and the smartphone 100-2 confirms the location at the time of shooting using the built-in GPS and the like. Send to the server 300. Alternatively, the smartphone 100-2 may generate location information in various ways, such as checking the location at the time of shooting using the location of the connected base station or AP (Access Point).

The server 300 determines whether an image capturing restriction element is included in the image using the received location information. A detailed method for determining whether the server 300 includes a shooting restriction element will be described below with reference to FIGS. 40 and 41.

On the other hand, if the photographing restriction element is included in the image, the server 300 corrects the image and transmits the modified image to the smartphone 100-2 of the user A (1).

Referring to FIG. 40, the server 300 analyzes location information received from the smartphone 100-2 of the user A (1) to check the location where the shooting is performed, and the identified location is where the shooting is restricted Judge whether or not. That is, the server 300 determines whether the identified shooting location is included in the previously stored shooting restriction location database. At this time, the location-limited location database stores location information that is restricted to the general public, such as a theater, a museum, or a military security area.

As a result of the determination, if the identified shooting location is included in the shooting restriction location database, the server 300 determines that the shooting restriction element is included in the image and corrects the image. In FIG. 40, the server 300 performs mosaic processing on the entire original image 4000a to generate a modified image 4000b.

Referring to FIG. 41, the server 300 analyzes location information received from the smartphone 100-2 of the user A (1) to check the location where the shooting is performed, and the confirmed location is where the shooting is restricted Judge whether or not. That is, the server 300 determines whether the identified shooting location is included in the previously stored shooting restriction location database. At this time, the location-limited location database stores location information that is restricted to the general public, such as a theater, a museum, or a military security area.

As a result of the determination, if the identified shooting location is included in the shooting restriction location database, the server 300 determines that the shooting restriction element is included in the image and corrects the image. In FIG. 41, the server 300 synthesizes a replacement image over the entire original image 4100a to generate a modified image 4100b. On the other hand, in addition, the server 300 may modify the image by blurring the entire image.

Meanwhile, in FIGS. 39 to 41, an embodiment in which the server 300 corrects an image is described, but differently, the server 300 merely analyzes the location information received from the smartphone 100-2 to set a location where the shooting is restricted. It is determined only whether it is applicable, and then transmits it to the smartphone 100-2, and the smartphone 100-2 receiving the same may modify the image.

42 is a flowchart illustrating a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.

Referring to FIG. 42, in step 4201, an image is captured on the user's device. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 4202, the user's device recognizes the location at the time of shooting, and transmits the location information to the server. At this time, the user's device can check the location at the time of shooting using the built-in GPS. Alternatively, the user's device may check the location at the time of shooting using the location of the connected base station or AP (Access Point).

In step 4203, the server determines whether the recognized location is included in the shooting restriction location. In detail, the server analyzes the received location information to check the location where the photographing was performed, and compares the identified location with a pre-stored photographic location database. At this time, the location-limited location database stores location information that is restricted to the general public, such as a theater, a museum, or a military security area.

As a result of the determination, if the identified shooting location is included in the shooting restriction location database, the server determines that the shooting restriction element is included in the image and proceeds to step 4204. In step 4204, the server modifies the image and transmits the modified image to the user's device. That is, the server generates a modified image by performing mosaic processing, blur processing, or alternative image synthesis processing on the image, and transmits the generated modified image to the user's device.

43 to 45 are diagrams for explaining a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention. Particularly, in the embodiments shown in FIGS. 43 to 45, the user device determines that the photographing restriction location corresponds to the photographing restriction position.

Referring to FIG. 43, the smartphone 100-2 of the user A 1 photographs an image including the user B 2 and the photographing prohibition sign 30. At this time, the photographing prohibition sign 30 may be composed of symbols and phrases meaning photographing prohibition, as illustrated in FIG. 43, or may be configured in other forms. For example, it may be configured in the form of a barcode, QR code, or color code, which includes the meaning of photographing prohibition. When the prohibition sign is composed of a barcode, QR code, or color code, the photographer cannot know that the corresponding sign corresponds to the prohibition sign, thereby preventing the photographer from intentionally avoiding the sign.

Referring to FIG. 44, the smartphone 100-2 of the user A 1 recognizes the photographing prohibition sign 30 included in the original image 4400a. In detail, the smartphone 100-2 recognizes at least one of the symbol 30-1 or the phrase 30-2 included in the photographing prohibition sign 30. The smartphone 100-2 that recognizes the photographing prohibition sign 30 determines that the image is photographed at the photographing restriction location. That is, the smartphone 100-2 determines that the shooting restriction element is included in the image. Accordingly, the smartphone 100-2 mosaics the entire original image 4400a to generate a modified image 4400b.

Referring to FIG. 45, the smartphone 100-2 of the user A 1 recognizes the photographing prohibition sign 30 included in the original image 4500a. In detail, the smartphone 100-2 recognizes at least one of the symbol 30-1 or the phrase 30-2 included in the photographing prohibition sign 30. The smartphone 100-2 that recognizes the photographing prohibition sign 30 determines that the image is photographed at the photographing restriction location. That is, the smartphone 100-2 determines that the shooting restriction element is included in the image. Accordingly, the smartphone 100-2 generates a modified image 4500b by synthesizing a replacement image over the entire original image 4500a.

46 is a flowchart illustrating a method of correcting an image captured at a location where photography is restricted according to an embodiment of the present invention.

Referring to FIG. 46, in step 4601, an image is captured on the user's device.

In step 4602, the user's device recognizes the objects included in the image.

In step 4603, the user's device determines whether a photographing restriction sign exists among the recognized objects. As a result of the determination, if there is a shooting restriction sign among the recognized objects, the process proceeds to step 4604, and the user device corrects the image. At this time, the user device modifies the image in a manner of mosaic processing, blur processing, or alternative image synthesis processing for the entire image.

47 to 49 are diagrams for explaining an embodiment of limiting photographing of a person according to privacy protection information according to an embodiment of the present invention.

Referring to FIG. 47, the smartphone 100-2 of the user A (1) photographs an image including the user B (2) and the user C (3). After shooting, the smartphone 100-2 of the user A (1) transmits the captured image to the server 300 and requests image processing. At this time, the smartphone 100-2 of the user A (1) may encrypt the captured image using the encryption key received from the server 300 and then transmit it to the server 300.

The server 300 determines whether an image capturing restriction element is included in the received image using the pre-stored privacy protection information 4710. If the received image is an encrypted image, the server 300 performs decryption of the image using an encryption key, and then determines whether a photographing restriction element is included. At this time, the privacy protection information 4710 is stored by mapping the time and location at which shooting is restricted for each user.

Meanwhile, a specific method of determining whether an image capturing restriction element is included in the image using the privacy protection information 4710 will be described in detail with reference to FIGS. 48 and 49 below.

When it is determined that the photographing restriction element is included in the received image, the server 300 corrects the image and transmits the corrected image to the smartphone 100-2 of the user A (1).

Hereinafter, with reference to FIGS. 48 and 49, the server 300 determines whether a shooting restriction element is included in the image using the privacy protection information 4710, and a detailed method of modifying the image will be described in detail.

Referring to FIG. 48, the smartphone 100-2 transmits the encrypted captured image to the server 300. At this time, the smartphone 100-2 also transmits location information on the location where the image is taken along with the image to the server 300. The smartphone 100-2 may use a built-in GPS or use the location of a connected base station or AP to determine the location where the shooting is performed and transmit it to the server 300 as location information. In the embodiment shown in FIG. 48, the smartphone 100-2 transmits "school" to the server 300 as location information for the photographing location.

Referring to FIG. 49, the server 300 decodes the received image and performs face recognition for the user B (2) and the user C (3) in the image. The server 300 checks the pre-stored privacy protection information 4910 to check the shooting time limit and the location corresponding to the recognized face user. In the privacy protection information 4910, at least one of a time or location at which shooting is restricted for each user is stored.

Looking at the privacy protection information 4910 shown in FIG. 49, the user A does not allow photographing between 24 and 06, and does not allow photographing at the “home”. User B does not allow shooting between 22:00 and 09:00, but the location where the shooting is restricted is not set. User C does not set a time limit for shooting, and does not allow shooting in “school”.

The server 300 determines whether an image capturing restriction element is included in the image based on the location information (“school”) received from the smartphone 100-2 of the user A 1 and the shooting time. To this end, the server 300 may simultaneously receive the time at which the photographing was performed when the photographed image is received from the smartphone 100-2 of the user A (1).

In FIG. 49, the server 300 uses the location information and time information received from the smartphone 100-2 of the user A (1), and the location and time at which the shooting was performed are “school” and “21 o'clock,” respectively. To confirm.

The server 300 checks the privacy protection information 4710 and confirms that the user B 2 is restricted from photographing between 22:00 and 9pm, and there is no photographing restriction condition for the location. The server 300 determines that the photographing time (“21 o'clock”) is not included in the photographing time limit of the user B 2, so the user B 2 determines that photographing is not restricted. That is, the server 300 determines that the user B 2 is not a shooting restriction element.

On the other hand, the server 300 refers to the privacy protection information 4710, and the user C 3 confirms that the shooting at the “school” is restricted, and that there is no shooting restriction condition for the time. The server 300 determines that the photographing location (“school”) is included in the photographing restriction location of the user C 3, and thus the user C 3 determines that photographing is restricted. That is, the server 300 determines that the user C 3 corresponds to the shooting limit element.

Therefore, the server 300 corrects the image by mosaicing only the user C (3).

50 is a flowchart illustrating an embodiment of limiting photographing of a person according to privacy protection information according to an embodiment of the present invention.

Referring to FIG. 50, in step 5001, an image including a person is photographed on a user device. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 5002, the user device encrypts the captured image using the encryption key received from the server. At this time, the encryption key is managed by the server.

In step 5003, the user device transmits the encrypted image and the location information of the device to the server.

In step 5004, the server decodes the received image and performs face recognition of the person included in the image. Specifically, face recognition may be performed by detecting a face region in an image and extracting features from the detected face region.

In step 5005, the server determines whether photographing of the recognized person is permitted by referring to pre-stored privacy protection information based on the location information and the shooting time received from the user device.

In step 5006, the server performs a protection process for a person who is not allowed to shoot. That is, the server may modify the image by performing at least one of mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing so that a person who is not allowed to shoot in the image cannot be identified.

In step 5007, the server transmits the modified image to the user device.

51 is a diagram showing the configuration of a user device according to an embodiment of the present invention. Referring to FIG. 51, the user device 1000 according to an embodiment of the present invention includes a user interface unit 1100, a photographing unit 1200, a storage unit 1300, a communication unit 1400, and a control unit 1500 You can. In addition, the control unit 1500 may include an image analysis unit 1510 and an image correction unit 1520.

On the other hand, the user device 1000 according to an embodiment of the present invention may be one of devices that can be photographed, such as smart glasses, smart watches, smart phones, tablets, cameras, or camcorders, as described above.

The user interface 1100 is a configuration that receives input from a user and displays a screen to the user, and may be implemented in various ways depending on the type of device. For example, if the user device 1000 is a smart phone, the user interface 1100 may include a touch screen.

The photographing unit 1200 may include a camera module embedded in a smart glass, a smart watch, a smartphone, or a tablet. The photographing unit 1200 may photograph a still image or a video, and transmit the photographed image to the control unit 1500.

The storage unit 1300 may include a memory in which photographed images are stored or various data are stored. For example, an address book, a picture, a video, or a record of use of a device may be stored in the storage unit 1300, and an image modified by the control unit 1500 may be stored.

The communication unit 1400 may perform data communication with other external devices or servers through wired or wireless.

The control unit 1500 is a component that controls the operation of the components included in the user device 1000, and in particular, analyzes the captured image to determine whether the photographing limiting element is included, and when the photographing limiting element is included Correct it. The control unit 1500 may include an image analysis unit 1510 and an image correction unit 1520.

The image analysis unit 1510 recognizes a location where a person, an object, or an image is captured in a captured image through image analysis, and at least one of a relation between a user of a device and a recognized person, a recognized object, or a recognized location Based on this, it may be determined whether or not the image needs to be corrected. In particular, the image analysis unit 1510 determines whether the user of the device and the recognized person are related to each other by using the data stored in the storage unit 1300, and if it is determined that it is not related, it is necessary to correct the image. You can.

For example, the image analysis unit 1510 checks whether the recognized person exists in the address book stored in the storage unit 1300, and if the recognized person does not exist in the address book, the user of the device and the recognized person are not related to each other. You can judge that.

Alternatively, the image analysis unit 1510 checks whether the recognized person exists in at least one picture stored in the storage unit 1300, and if the recognized person does not exist in the stored at least one picture, the user of the device and the recognized person It can be judged that these are not related.

Alternatively, the image analysis unit 1510 analyzes the SNS (Social Networking Service) usage record of the user of the device using the data stored in the storage 1300 to determine whether the user of the device and the recognized person are related to each other. can do.

Alternatively, the image analysis unit 1510 may access the database of the external server through the communication unit 1400 to determine whether the photographing of the recognized object or location is restricted, and if the photographing is restricted, it may be determined that image correction is necessary. have.

In addition, when the image analyzing unit 1510 recognizes a photographing prohibition sign from the photographed image, the image analysis unit 1510 may determine a location where the image is photographed as a location where photographing is restricted, and thus, may need to correct the image.

Meanwhile, the image correction unit 1520 is configured to perform image correction when it is determined that the image analysis unit 1510 needs to correct the image. The image correction unit 1520 may perform at least one of mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing on the entire person, object, or image included in the image. For detailed examples of the image correction unit 1520 correcting an image, refer to FIGS. 7 to 12 described above.

Meanwhile, the components included in the user device 1000 may perform operations for implementing the content playback control method described with reference to FIGS. 1 to 50, even though the operation is not described with reference to FIG. 51. .

Meanwhile, a point in time at which the captured image is modified may be set in various ways. For example, an image may be corrected after shooting and before storing it in a user terminal. Alternatively, the original image is stored in the user terminal, and the image may be modified when the stored image is transmitted to an external server or another terminal. Alternatively, an image may not be stored in a user terminal performing shooting, and the captured image may be directly modified and transmitted to an external server or another terminal. Hereinafter, various embodiments in which the viewpoint of modification of the image is different will be described.

52 is a view for explaining an embodiment of modifying a captured image before storing the captured image in a user terminal in a method of modifying the captured image according to an embodiment of the present invention.

On the left side of FIG. 52, a gallery app is executed on the smartphone 100-2 to show a plurality of images displayed on the screen, and when a single image 5210 is selected by touching one of the plurality of images, on the right side As illustrated, the selected image 5210 is displayed as an enlarged image 5220.

It is understood that the selected image 5210 among the plurality of images displayed on the left side of FIG. 52 includes a person whose shooting is restricted, and thus the area 5211 corresponding to the face of the person whose shooting is restricted is mosaicized. You can. Also, in the enlarged image 5220 displayed on the right side, the area 5221 corresponding to the face of the person whose shooting is restricted is mosaic-processed.

Before capturing an image on the smartphone 100-2, and saving the captured image on the smartphone 100-2, it is determined whether there is a shooting limit element in the captured image, and if it is determined that there is a shooting limit element, the image After correction, the corrected image is stored in the smartphone 100-2. That is, since the original image is not stored in the smartphone 100-2, only the modified image can be checked even if the stored image is accessed on the smartphone 100-2. Accordingly, according to the present embodiment, even the user of the smartphone 100-2 can access the modified image only.

For a detailed method of determining whether a shooting restriction element is included and correcting an image, refer to the foregoing.

53 is a flowchart for explaining an embodiment of modifying a captured image before storing the captured image in a user terminal in a method of modifying the captured image according to an embodiment of the present invention.

Referring to FIG. 53, in step 5301, an image is captured on the user's device. At this time, the user's device may be a variety of devices capable of shooting, such as a smartphone, camera, or tablet.

In step 5302, the user's device recognizes at least one of a person, an object, or a location where the image is captured in the captured image. At this time, for a detailed method of recognizing the location of the person, object, or image captured in the image, refer to the bar described with reference to FIG. 2 above.

In step 5303, the user's device determines whether an image capturing restriction element is included in the image based on at least one of a relationship between a user of the device and a recognized person, a recognized object, or a recognized location. At this time, for a detailed method of determining whether to include the shooting restriction element, refer to the bar described with reference to FIG. 2 above.

In step 5304, the user's device determines whether or not the photographing restriction element is included in the image, and if so, proceeds to step 5305 to correct the photographed image and stores the modified image in the user's device. However, if the result of the determination does not include the shooting restriction element, the process proceeds to step 5306, and the user's device stores the captured image without modification.

54 and 55 are diagrams for explaining an embodiment of modifying an image stored in a user terminal when transmitting it to the outside in a method of correcting a captured image according to an embodiment of the present invention.

On the left side of FIG. 54, a gallery app is executed on the smartphone 100-2 to show a state in which a plurality of images are displayed on the screen, and when any one of the plurality of images is touched and selected, the right side is selected. As shown, the selected image 5410 is displayed as an enlarged image 5420.

The selected image 5410 among the plurality of images displayed on the left side of FIG. 54 includes a person whose shooting is restricted, but the area 5161 corresponding to the face of the person whose shooting is restricted is not mosaic-processed. Able to know. In addition, even in the enlarged image 5420 displayed on the right side, mosaic processing is not performed on the area 5221 corresponding to the face of the person whose shooting is restricted.

In this embodiment, since the smartphone 100-2 stores the captured image without modification, the user can access the original unmodified image through the gallery app of the smartphone 100-2.

However, in the present exemplary embodiment, when the image stored in the smartphone 100-2 is transmitted to an external server or other device, it is determined whether the stored restriction element is included in the stored image, and if it is included, the image is corrected Transfer to the outside.

Referring to FIG. 55, a user selects any one 5510 from a plurality of images displayed in the gallery app of the smartphone 100-2, and selects a send button 5530 for sharing the selected image 5510. If it does, the smartphone 100-2 determines whether the photographing restriction element is included in the stored original image, and if so, corrects the image and transmits the modified image to the outside.

In FIG. 55, the smartphone 100-2 transmits the modified image 5520 in which the area 5521 corresponding to the face of the person whose shooting is restricted is mosaicized to the server 300 designated as a destination.

56 is a flowchart for explaining an embodiment of modifying an image stored in a user terminal when transmitting it to the outside in a method of modifying a captured image according to an embodiment of the present invention.

Referring to FIG. 56, in step 5601, an image is captured on the user's device. At this time, the user's device may be a variety of devices capable of shooting, such as a smartphone, camera, or tablet.

In step 5602, the user device stores the captured image in a memory inside the user device.

In step 5603, the user device receives a request to transmit the stored image from the user to an external server or another device.

In operation 5604, the user device recognizes at least one of a person, an object, or a location where the image is captured in the stored image. At this time, for a detailed method of recognizing the location of the person, object, or image captured in the image, refer to the bar described with reference to FIG. 2 above.

In step 5605, the user device determines whether an image capturing restriction element is included in the image based on at least one of a relationship between a user of the device and a recognized person, a recognized object, or a recognized location. At this time, for a detailed method of determining whether to include the shooting restriction element, refer to the bar described with reference to FIG. 2 above.

In step 5606, the user device determines whether or not the photographing restriction element is included in the stored image, and if so, proceeds to step 5607 to correct the stored image and transmit the modified image to the outside. However, if the result of the determination does not include the shooting restriction element, the process proceeds to step 5608, and the user device transmits the stored image to the outside without modification.

FIG. 57 is a view for explaining an embodiment in which a method of correcting a captured image according to an embodiment of the present invention transmits the captured image to the outside without directly storing the captured image in a user terminal.

Fig. 57 shows two smartphones 100-2a and 100-2b performing video calls with each other.

When making a video call, the smartphones 100-2a and 100-2b do not save the video call screen unless there is a special request. However, even in the case of a video call, since there is a need to correct the shooting limiting element, in this embodiment, the smartphone 100-2a directly modifies the captured image without saving and transmits it to another smartphone 100-2b. . Referring to the video call screen 5710 displayed on the smartphone 100-2b, no modification is made to the face area 5712 of the party of the video call, but the face area 5711 of the person whose shooting is restricted is mosaic It can be seen that it has been processed.

58 is a flowchart for explaining an embodiment in which a captured image is directly modified and transmitted to the outside without storing the captured image in the user terminal in the method of correcting the captured image according to an embodiment of the present invention.

Referring to FIG. 58, in step 5801, an image is captured on the user's device. At this time, the user's device may be a variety of devices capable of shooting, such as a smartphone, camera, or tablet.

In step 5802, the user's device recognizes at least one of a person, an object, or a location where the image is captured in the captured image. At this time, for a detailed method of recognizing the location of the person, object, or image captured in the image, refer to the bar described with reference to FIG. 2 above.

In step 5803, the user's device determines whether an image capturing restriction element is included in the image based on at least one of a relationship between a user of the device and a recognized person, a recognized object, or a recognized location. At this time, for a detailed method of determining whether to include the shooting restriction element, refer to the bar described with reference to FIG. 2 above.

In step 5804, the user's device determines whether or not the photographing restriction element is included in the image, and if so, proceeds to step 5805 to correct the photographed image and transmit the modified image to an external server or another device. . However, if the result of the determination does not include the shooting restriction element, the process proceeds to step 5806, and the user's device transmits the captured image to an external server or other device without modification.

59 to 61 are diagrams for explaining an embodiment in which a person whose photographing is restricted at a specific location modifies the photographed image according to another embodiment of the present invention.

Referring to FIG. 59, the smartphone 100-2 of the user A (1) is allowed to photograph (for example, a street, a restaurant, etc.) 5910a for a same person 5911 or a place where photographing is restricted ( For example, an image of a theater, museum, or military security area) 5920a may be taken. Whether or not the same person 5911 may be restricted depending on the shooting position. For example, in the case of a celebrity, photography at a venue is restricted due to copyright problems, but photography on the street may not be restricted. The user's (A1) smartphone 100-2 transmits the captured image to the server 300 to request image processing. At this time, the smartphone 100-2 of the user A (1) transmits location information to the server 300 together with the captured image. The transmitted location information is information about the location where the smartphone 100-2 has taken the photograph, and the smartphone 100-2 confirms the location at the time of shooting using the built-in GPS and the like. It can be transmitted to the server 300. Alternatively, the smartphone 100-2 may generate location information in various ways, such as checking the location at the time of shooting using the location of the connected base station or AP (Access Point).

Referring to FIG. 60, the server 300 analyzes location information received from the smartphone 100-2 of the user A (1) to check the location where the shooting was performed, and the identified location is included in the image It may be determined whether the photographing of the camera corresponds to a location where the photographing is restricted. That is, the server 300 determines whether the photographed location identified in the photographed restricted location database stored in advance is included as a location where photographing for a person included in the image is restricted. At this time, the location where the shooting is restricted for each user may be stored in the shooting restriction location database, or location information for which the shooting of the general public is restricted, such as a performance hall, a museum, or a military security area, may be stored.

As a result of the determination, if the photographing location where the original image 5910a is photographed is not a location where photographing of a person included in the image is not restricted, the server 300 may transmit the uncorrected image 5910b to the user's device. Alternatively, the server 300 may transmit a message that the image does not need to be modified to the user's device.

On the other hand, as a result of the determination, if the photographing location where the original image 5920a is photographed is a location where photographing of a person included in the image is restricted, the server 300 determines that the photographing restriction element is included in the original image 5920a. do. Also, the server 300 may modify a part of the image corresponding to the person whose shooting is restricted. Specifically, the smartphone 100-2 outputs a corrected image 5920b by performing mosaic processing on the face area of the person 5911 photographing is restricted from the original image 5920a, or by synthesizing or blurring an alternative image. can do.

On the other hand, in FIGS. 59 to 60, an embodiment in which the server 300 modifies an image is described, but differently, the server 300 analyzes the location information received from the smartphone 100-2 and sets the location in the restricted location. It is determined only whether it is applicable, and then transmits it to the smartphone 100-2, and the smartphone 100-2 receiving the same may modify the image.

On the other hand, in FIGS. 59 to 60, an exemplary embodiment in which a location in which photographing is restricted is applied differently according to an individual to be photographed has been described, but differently, a location in which photographing is restricted for all users may be equally applied. In other words, regardless of the person included in the captured image, the location of the restricted image may be determined, and if it is determined that the image is in the restricted location, a part of the image corresponding to the person or object included in the image may be corrected.

Alternatively, as shown in FIG. 61, the smartphone 100-2 of the user A 1 analyzes the identified location information to confirm the location where the shooting was performed, and the identified location is located at a location where the shooting is restricted. You can also determine if it is applicable. That is, the smartphone 100-2 determines whether the identified shooting location is included in the shooting restriction location database of the storage unit 1300 of the smartphone 100-2.

As a result of the determination, if the photographing location where the original image 5910a is photographed is not a location where photographing of a person included in the image is not restricted, the smartphone 100-2 may store the image 5910b where the image is not corrected. .

On the other hand, as a result of the determination, if the photographing location where the original image 5920a is photographed is a location where photographing of a person included in the image is restricted, the smartphone 100-2 includes a photographing restriction element in the original image 5920a. I think there is. In addition, the smartphone 100-2 may modify a part of the image corresponding to the person whose shooting is restricted. Specifically, the smartphone 100-2 performs mosaic processing on the face region of the person 5911 with limited spirit in the original photographed image 5920a, synthesizes an alternative image, or blurs the image to store the corrected image 5920b. You can.

As described above, when photographing a person or object, photographing of the person or object may be restricted or photographing of the person or object may not be limited according to a photographing position where the person or object is photographed.

62 is a flowchart illustrating a method of correcting a photographed image at a location where photographing is restricted according to an embodiment of the present invention.

Referring to FIG. 62, in step 6201, an image is captured on the user's device. At this time, any one of a general shooting device such as a camera or a camcorder, a wearable device such as a smart glass or a smart watch, or a mobile device such as a smartphone or tablet may be used for shooting, and various devices capable of shooting may be used.

In step 6202, the user's device recognizes the location at the time of shooting. At this time, the user's device can check the location at the time of shooting using the built-in GPS. Alternatively, the user's device may check the location at the time of shooting using the location of the connected base station or AP (Access Point).

In step 6203, the server determines whether the recognized location is included in the shooting restriction location. Alternatively, it may be determined whether the location where the user's device is recognized is included in the photographing restriction location. Specifically, it may be determined whether the photographing of a person included in the image corresponds to a location where the photographing is restricted or a location where the photographing is restricted regardless of the subject. To this end, the server or the user's device analyzes the received location information to check the location where the shooting was performed, and compares the identified location with a pre-stored restricted location database. At this time, the location information for which photography is restricted for each person is stored in the location-restricted location database, or location information for which photography of the general public is restricted, such as a performance hall, a museum, or a military security area, is stored.

As a result of the determination, if the recognized shooting position corresponds to the shooting limit position, in step 6204, the server or the user's device may determine whether there is a shooting limit element in the image. That is, if the recognized shooting position is a position where the shooting for a specific person is restricted, whether a specific person is restricted, or if the recognized shooting position is a position where shooting is restricted to all persons or objects, at least one person or It is possible to determine whether an object is included. If the image capture restriction element is included, the server may modify the image in step 6205 and transmit the modified image to the user's device. Alternatively, the user's device may modify the image and store the modified image. For example, the server or the user's device may generate a modified image by performing mosaic processing, blur processing, or alternative image synthesis processing on the image.

63 to 66 are diagrams for explaining an embodiment of modifying a photographed image according to a user's selection according to an embodiment of the present invention.

Referring to FIG. 63, an image including the user B (2) and the user C (3) is photographed on the smartphone 100-2 of the user A (1). Subsequently, the smartphone 100-2 of the user A 1 displays the captured image on the screen.

64 and 65 respectively show examples in which a part of a corresponding image of a selected person is modified when a user selects a person to perform a protection process from a captured image by a different method. Here, the situation in which the user selects a person to perform the protection process, for example, before the photographed image is stored in the storage unit of the smartphone 100-2 or shared with an external device, the smartphone 100- 2) In the state of being provided as a preview screen or a temporary screen on the screen, it may be a situation in which the user selects a person to perform protection processing on the captured image.

Referring to (a) of FIG. 64, the smartphone 100-2 recognizes the faces of the user B (2) and the user C (3) by performing face recognition on the captured original image 6400a. Subsequently, the smartphone 100-2 may provide recognition results indicating that the faces of the users B 2 and C 3 are recognized. The recognition result may be, for example, the display of the faces or outlines of the faces of the user B (2) and the user C (3) recognized as visual feedback by highlighting (eg, blinking or coloring). Subsequently, when the user selects one of the user B (2) and the user C (3) (C (3)), the smartphone 100-2 can perform the protection process for the selected user C (3). have. For example, the smartphone 100-2 may modify a part of the image corresponding to the user C 3 so that the selected user C 3 cannot be identified. In detail, as shown in (b) of FIG. 64, the smartphone 100-2 performs mosaic processing on the face region of the user C (3), synthesizes alternative images, or blurs the processed image ( 6400b) can be displayed on the screen.

As another embodiment, referring to (a) of FIG. 65, the smartphone 100-2 recognizes the faces of the user B (2) and the user C (3) by performing face recognition on the captured original image. Subsequently, the smartphone 100-2 may determine whether the recognized face is a person whose shooting is restricted. As a result of determination, if the user B (2) and the user C (3) are persons whose shooting is restricted, the smartphone 100-2 may perform protection processing for the user B (2) and the user C (3). . For example, the smartphone 1002 displays a portion of the image corresponding to the user B (2) and a portion of the image corresponding to the user C (3) so that the identification of the user B (2) and the user C (3) is impossible. Can be modified. In detail, the smartphone 100-2 performs mosaic processing on the face regions of the user B (2) and the user C (3), which are persons whose shooting is restricted, or synthesizes or substitutes an alternate image or blurs the modified image ( 6500a) on the screen. Subsequently, when the user selects one of the user B (2) and the user C (3) (B (2)), the smartphone 100-2 is a person to release the protection process of the selected user B (2). I can judge. Subsequently, the smartphone 100-2 may modify a part of the image corresponding to the user B 2 so that the selected user B 2 can be identified. In detail, as shown in (b) of FIG. 65, the smartphone 100-2 releases mosaic processing on the face region of the user B (2), releases synthesis of an alternative image, or releases blur processing. The modified image 6500b can be displayed on the screen.

As another embodiment, referring to (a) of FIG. 66, the smartphone 100-2 performs face recognition on the captured original image 6600a to display the faces of the users B (2) and C (3). Recognize. Subsequently, the smartphone 100-2 may provide a pop-up screen 6610 asking whether to perform the protection process for the recognized user C 3. Through the pop-up screen 6610, when the user selects a button requesting to perform protection processing, the smartphone 100-2 may perform protection processing on the user C (3). And, as shown in (b) of Figure 66, the smartphone 100-2 is a modified image (6600b) by mosaic processing on the face region of the user C (3), or by synthesizing alternative images or blur processing ) Can be displayed on the screen.

The screen through which the user can select a person to perform the protection process will be displayed after the image is taken and before the smartphone 100-2 stores the captured image in the storage unit of the smartphone 100-2. You can. Alternatively, it may be displayed before the smartphone 100-2 provides the captured image to an external device (eg, a server or a third-party smartphone).

Meanwhile, when a person to be protected is selected by the user, information on the selected person may be stored in a database of the smartphone 100-2 or the server 300. For example, a picture of a selected person can be stored as a shooting limit element. In this case, if there is a person who has been protected by the user in a later photographed image, the smartphone 100-2 may automatically determine the person whose protection has been performed as a person whose shooting is restricted. In addition, the smartphone 100-2 may perform mosaic processing on an area corresponding to the person, synthesize a substitute image, or perform blur processing.

As such, the method for the user to select a person to limit shooting may be more useful in a situation in which the smartphone 100-2 or the server 300 does not have information on the recognized face of the captured image. That is, when there is no information on the recognized face, the image is determined by determining the processing of the recognized face by the user's selection without performing uniform processing such as limiting or canceling shooting of the recognized face. The degree of freedom of the processing user can be further increased.

67 and 68 are flowcharts illustrating a method of correcting a photographed image according to a user's selection according to an embodiment of the present invention.

Referring to FIG. 67, in step 6701, an image including a person is photographed on the user's device. At this time, the user's device may be a variety of devices capable of shooting, such as a smartphone, camera, or tablet.

In step 6702, the user's device performs face recognition of the person included in the captured image.

In operation 6703, the user's device may provide a recognition result indicating that the face is recognized in the captured image. For example, the user's device may display a screen with visual feedback applied to the recognized person.

In a state in which the recognition result screen is displayed in step 6704, the user's device may select a person who needs protection from the user.

If a person requiring protection is selected by the user, in step 6705, the user device may perform protection processing for the selected person. For example, the user device may modify the image so that the selected person cannot be identified in the image.

As another embodiment, referring to FIG. 68, in step 6801, an image is captured on the user's device.

In step 6802, the user's device performs face recognition of the person included in the captured image.

In step 6803, the user's device may determine whether the recognized face is a person whose shooting is restricted. As a result of the determination, if the recognized person is a person whose shooting is restricted, in step 6804, the user's device may perform a protection process on the person whose shooting is restricted. As a result of the determination, if the recognized person is a person whose shooting is not restricted, the user's device may not perform the protection process.

In step 6805, the user's device may display a person who has been protected. For example, the user's device may perform mosaic processing on a region corresponding to a recognized person in the captured image, synthesize alternate images, or perform blur processing.

In step 6806, while the person to whom the protection process has been performed is displayed, the user's device may select a person to cancel the protection process from the user.

If a person to cancel the protection process is selected, in step 6807, the user's device may cancel the protection process for the person to whom the protection process has been performed. For example, the user's device may release mosaic processing on an area corresponding to a selected person from the photographed image, release synthesis of an alternative image, or cancel blur processing.

So far, the present invention has been focused on the preferred embodiments. Those skilled in the art to which the present invention pertains will appreciate that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be construed as being included in the present invention.

Meanwhile, the above-described embodiments of the present invention can be written in a program executable on a computer, and can be implemented on a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium includes, for example, magnetic storage media such as a ROM, floppy disk, or hard disk, optical read media such as CD-ROM, DVD, etc., and storage media such as carrier waves such as transmission via the Internet. .

1000: user device 1100: user interface
1200: recording unit 1300: storage unit
1400: communication unit 1500: control unit

Claims (20)

For a user device,
A storage unit in which data is stored; And
If a person photographed by the user device or included in an image received from the outside by the user device is recognized, and the recognized person is not related to a user of the user device by using data stored in the storage unit , A control unit for correcting a part of the image corresponding to the recognized person so that the recognized person cannot be identified,
The control unit determines whether a location where the image is photographed is a location where photographing is restricted, and when the photographing is restricted, a user device that replaces the image with another image. The method of claim 1, wherein the user device,
A user device further comprising a photographing unit for photographing the image. delete According to claim 1,
When the data stored in the storage unit is an address book,
The control unit determines that the user and the recognized person are not related to each other if the recognized person does not exist in the address book. According to claim 1,
When the data stored in the storage unit is at least one picture,
The control unit determines that the user and the recognized person are not related to each other if the recognized person does not exist in the at least one picture. According to claim 1,
The control unit determines whether the user and the recognized person are related to each other by analyzing the user's social networking service (SNS) record using the data stored in the storage unit. According to claim 1,
The control unit accesses the database of the external server to determine whether the location where the image is captured is a location where the shooting is restricted. According to claim 1,
When the control unit recognizes a shooting restriction object in the image, the user device characterized in that it determines the location where the image is captured as a location where the shooting is restricted. According to claim 1,
When it is determined that a part of the image needs to be corrected, the control unit performs at least one of mosaic processing, blur processing, background screen synthesis processing, or alternative image synthesis processing on a part of the video. . In the system for modifying the image,
A user device that transmits the image to a server and requests image processing; And
If a person included in the image received from the user device is recognized, and the recognized person is not related to the user of the user device, a part of the image corresponding to the recognized person is impossible to identify the recognized person Correcting, transmitting the corrected image to the user device,
Using the pre-stored data, it is determined whether the location where the image is photographed is a location where the photographing is restricted, and if the photographing is restricted, the image is replaced with another image, and the user device is replaced with the other image. System comprising a server that transmits the video. delete The method of claim 10,
The server receives the time and location at which the image was taken from the user device, and determines whether or not shooting of the recognized person is restricted at the received time and location using pre-stored data. . The method of claim 10,
The user device encrypts the image using the encryption key received from the server and transmits it to the server,
The server decodes the received image and then corrects the image corresponding to the person included in the image. For a user device,
A photographing unit for photographing an image;
When performing shooting using the photographing unit, a photographing notification is transmitted to a photographed device, and a photograph of the photographed subject and a photographing permission message are received from the photographed device as photographing is not permitted by the photographed photograph Communication department; And
As the message indicating whether or not the shooting is allowed indicates a non-shooting message, a face recognized in the image photographed through the photographing unit is compared with the received photograph of the subject, and a face matching the photograph of the subject in the photographed image The user device including a control unit for modifying the captured image to protect the process. In the method of modifying the image by the user device,
Recognizing a person included in the image;
Determining whether the recognized person is related to a user of the user device;
Modifying a part of the image corresponding to the recognized person so that the recognized person cannot be identified if the recognized person is not related to the user;
Determining whether a location where the image is photographed is a location where photographing is restricted; And
And replacing the image with another image if the location where the image is photographed is a location where photography is restricted. delete The method of claim 15,
Determining whether or not the recognized person is related to the user,
And when the recognized person does not exist in the address book stored in the user device, determining that the user and the recognized person are not related to each other. The method of claim 15,
Determining whether or not the recognized person is related to the user,
And if the recognized person does not exist in at least one picture stored in the user device, determining that the user and the recognized person are not related to each other. The method of claim 15,
Determining whether or not the recognized person is related to the user,
A method characterized by determining whether the user and the recognized person are related to each other by analyzing the user's social networking service (SNS) usage record using data stored in the user device. The method of claim 15,
Step of correcting a part of the image,
And performing mosaic processing, blur processing, wallpaper synthesis processing, or alternative image synthesis processing on a part of the image.

Images