TW201618033A - Method of modifying image including photographing restricted element, and device and system for performing the method - Google Patents

Abstract

Provided is a user device including: a storage configured to store data; and a controller configured to recognize a person included in an image captured by the user device or received by the user device from an external source, and change a part of the image corresponding to the recognized person to be unrecognizable in response to a degree of closeness between the recognized person and a user of the user device being lower than a predetermined level based on the data stored in the storage.

Description

Image correction method including photographing restriction element, device and system for performing the same

Apparatus and method consistent with an exemplary embodiment relates to a method of correcting an image (where a photo or moving image is captured by a user device, or a moving image is corrected), and a device and a method for performing Method of the system.

When a photo or moving image is captured by using a photographing device such as a video camera or a video camera, a stranger can be included in the captured photo or the captured moving image. At this point, the stranger may be exposed to the captured photo or captured moving image against his/her wishes and, in some cases, may infringe the privacy of the stranger. However, it is difficult for a photographer to notice a stranger whenever he/she is photographing.

When the photography restriction individual is classified from the people included in the captured image and the image processing is performed so that the photography restriction individual is indistinguishable, the photographer can protect the privacy of the photographer while conveniently using the general method. Capture images.

In particular, in recent years, since a portable device such as a smart phone or a personal computer (PC) generally supports a photographing function, an individual may most likely be photographed by a stranger, and thus preparation is necessary.

At the same time, the photographic limiting element can include not only an individual but also an item or location. Therefore, if the image captured by the personal device includes a photographing restriction object or a photographing restriction position, image processing can be performed such that the photographing restriction object or the photographing restriction position is indistinguishable.

If the image captured by the personal device includes a photography limiting element, one or more of the illustrative embodiments provide a method of correcting the image.

According to an exemplary embodiment, a user device is provided, comprising: a memory configured to store data; and a controller configured to recognize that the device is captured by a user device or by a user device The external source receives the individual in the image, and based on the data stored in the storage, the degree of intimacy between the identified individual and the user device user is lower than a predetermined level and corresponds to the recognized personal image. Some of them changed to be unrecognizable.

The user device can further include a configuration to capture the video camera.

The controller can be further configured to determine if the location of the captured image is a photographic limited position and to change a portion corresponding to the recognized personal image to be unrecognizable when the location is a photographic limited position.

The data stored in the storage can be an address book, and the controller can be further configured to determine that the degree of intimacy is below a predetermined level in response to the identified individual not being included in the address book.

The data stored in the storage may be at least one photo, and the controller may determine that the degree of intimacy between the user and the identified individual is lower than a predetermined level in response to the identified individual not included in the at least one photo .

The data stored in the storage can be a user social network service (SNS) usage record, and the controller can determine the degree of intimacy by analyzing the SNS usage record.

The controller can determine whether the position of the captured image is the photographing limit position by accessing the external server database, and correct the image when the position is determined to be the photographing limit position.

The controller can be further configured to identify the photographic restriction object from the image and can determine the location of the captured image as the photographic limited position in response to the image recognition photographic restriction object.

The controller may perform at least one of a mosaic process, a blur process, a background image layout process, and an alternate image layout process on a portion of the image to change a portion of the image to be unrecognizable.

In accordance with another aspect of the exemplary embodiments, a system for correcting images is provided, configured to execute an image processing server, and configured to request image processing while transmitting images to a server user device, The server is further configured to identify an individual included in the image received from the user device and in response to the degree of intimacy between the identified individual and the user device user being below a predetermined level, corresponding to the identified Part of the personal image was changed to be unrecognizable.

The server can determine whether the position of the captured image is the photographing limit position by using the pre-stored data, and change the portion corresponding to the recognized personal image to be unrecognizable in response to the position being the photographing limit position.

The server can receive the time and location of capturing the image from the user device, and determine whether the recognized personal photography is restricted at the received time and the received location by using the pre-stored data.

The user device can encode the image by using the self-server to receive the encryption key and transmit the encoded image to the server, and the server can decode the encoded image and correct the portion corresponding to the recognized individual decoded image.

According to another aspect of the exemplary embodiments, a user device is provided, comprising: a camera configured to capture an image; a communication interface configured to transmit a photography notification to a device captured by the image in the image And receiving a response to the photographic notification from the subject's device; and a controller configured to correct the image based on the indication of whether the captured image is permitted to respond.

According to another aspect of the exemplary embodiments, a method for correcting an image by a user device includes: identifying an individual included in the image; determining a degree of intimacy between the identified individual and the user device user Whether it is lower than the predetermined level; and changing a portion corresponding to the recognized personal image to be unrecognizable in response to the degree of intimacy being lower than the predetermined level.

The determining may include determining whether the position of the captured image is a photographing limit position, and the changing may include modifying the portion of the image when the position is the photographing limit position.

The determining may include determining that the degree of intimacy is below a predetermined level in response to the identified individual not being included in the address book stored in the user device.

The determining may include determining that the degree of intimacy is below a predetermined level in response to the identified individual not being included in the at least one photo stored in the user device.

The determining may include determining whether the degree of intimacy is below a predetermined level by analyzing a user social network service (SNS) usage record based on the data stored in the user device.

The changing may include performing at least one of a mosaic process, a blurring process, a background image layout process, and an alternate image layout process on a portion of the image to change a portion of the image to be unrecognizable.

According to another aspect of the exemplary embodiments, a method for correcting an image by a server includes: receiving an image and a video processing request from a user device; and identifying an object included in the image in response to the image processing request; Determining whether the object is a photographing restriction element by at least one of capturing a position of the image, a time of capturing the image, and a degree of intimacy between the user and the recognized object; and executing the image in response to the object being determined to be a photographing restriction element Process to correct the objects contained in the image.

The image processing request may indicate the type of image processing to be performed on the object, and the image processing type may correspond to mosaic processing, blur processing, background image processing, or alternative image editing processing.

The image processing request may include at least one of a location at which the image was captured, a time at which the image was captured, and a degree of intimacy between the user and the identified object (location).

Determining whether the object is a photography limiting component can include analyzing the received image to extract features indicative of location information, comparing the extracted features to data stored in the server to find a location; and responding to the location designated as a photography prohibited region The object is determined to be a photography limiting element.

The illustrative embodiments are described in more detail below with reference to the accompanying drawings.

In the following description, like reference numerals are used for the The matters defined in the description, such as the detailed description, However, it will be apparent that the exemplary embodiments can be practiced without the specific items defined. Further, well-known functions or constructions are not described in detail, as well-known functions or constructions will confuse the description with unnecessary details.

When preceded by a list of components, an expression such as "at least one of" modifies the entire list of elements and does not modify the individual elements of the list.

FIG. 1 is a diagram for describing an environment for performing a method of correcting an image including a photographing restriction element, according to an exemplary embodiment.

Referring to FIG. 1, user A can capture an image containing a person or an object, or an image of a specific location, by using the smart glasses 100. Here, the captured image may be not only a still image such as a photo, but also a moving image.

As shown in FIG. 1, the smart glasses 100 can capture images containing individuals (such as user B), objects (such as statues 20), or photographic signs 30. The smart glasses 100 can identify at least one of an individual included in the image, an object included in the image, and a position at which the image is captured, and can determine, based on the result of the recognition, whether the image includes a component that restricts photography (ie, photography) Limiting components).

Here, the photography restriction element limits the person, object or location to be photographed due to privacy, copyright or security. This photographic limiting element can be applied consistently to all people or to different applications depending on the photographer. For example, photography in a military security zone is limited to all people, but photography of a particular individual may be limited to individuals who do not know a particular individual. If the photographing restriction person or object is included in the image or the image is captured at the photographing restriction position, it is determined that the image includes the photographing restriction element.

Meanwhile, when the smart glasses 100 capture images and transmit the images to a device or server connected to the smart glasses 100, the device or the server can recognize the position of the person, the object or the image and determine whether the image contains the photographing restriction element.

If it is determined that the image includes the photographing restriction element, the smart glasses 100 or the device or the server connected to the smart glasses 100 corrects the image. In particular, the image is corrected so that the photography limiting element is indistinguishable or unrecognizable. The method of correcting the image will be described in detail later.

Meanwhile, the user A can capture an image by using any of various devices other than the smart glasses 100. For example, user A may use a wearable device (such as smart watch 100-1), a mobile device (such as smart phone 100-2 or tablet personal computer (PC) 100-5), or a general photographic device (such as Camera 100-3 or camcorder 100-4). However, any of a variety of devices capable of capturing images or moving images is usable and will be apparent to those of ordinary skill in the art.

The device that is typically carried by most users is smart phone 100-2, and information about the user (which needs to perform one or more exemplary embodiments) is highly likely to be stored in smart phone 100-2. Further, since the smart phone 100-2 can correct the image, it is assumed that the device described below by the user is the smart phone 100-2.

2 is a flow chart of a method of modifying an image containing a photographic limiting element, in accordance with an illustrative embodiment.

Referring to FIG. 2, in operation 201, an image is captured by using a user device. Here, the image can be obtained by using any of a general photographing device (such as a camera or a video camera), a wearable device (such as a smart glasses or a smart watch), and a mobile device (a smart phone or a tablet PC). Or capture by using any other variety of devices that can be photographed.

In operation 202, the user device identifies at least one of an individual included in the image, an object included in the image, and a location at which the image is captured. Here, the user device can recognize the person included in the image by using, for example, a face recognition method. However, the exemplary embodiments are not limited thereto, and the user device can recognize the person included in the image by using any of various methods for discriminating the individual. The facial recognition method includes extracting features from the image detecting face and self-detecting the face.

At the same time, the user device can recognize the object included in the image by detecting the object included in the image and extracting features such as the shape and color of the detected object.

At the same time, the user device can recognize the position of the captured image by using a method of recognizing the absolute position, a method of determining whether the position is a specific place, and a method of determining whether the position is a photographing limit position.

First, the method of identifying the absolute position can be captured by recognizing a Wi-Fi-based positioning system (WPS) via a global positioning system (GPS) or embedded in a user device. The absolute position (such as latitude and longitude) at the time point of the image is performed.

Second, the method of determining whether the location is a specific location can be performed by determining whether the user device is located at a particular location (eg, a museum, restaurant, or church) at the point in time when the image was captured. Alternatively, the absolute location of the user device can be identified by using the GPS of the user device, and the absolute location of the user device can be compared to a particular location, which is pre-stored in the database.

Again, the method of determining whether the position is the photographing limit position can be performed by determining whether the user device is at the photographing limit position at the time point of capturing the image. For example, the user device can determine whether the location is a forbidden photographic area by analyzing a flag included in the image, such as a photographic area-inhibited flag. Alternatively, the absolute position of the user device can be identified by using the GPS of the user device, and the absolute position of the user device can be compared with the absolute position of the prohibited shooting area, and the shooting area is prohibited from being pre-stored in the database.

In operation 203, after identifying at least one of the person, the object, and the location, the user device is based on the relationship between the user of the user device and the identified individual, the identified object, and the identified location At least one of the images determines whether the image contains a photographing restriction element. In other words, the user device determines whether the image needs to be corrected. If the image includes a photographing restriction element, the user device determines that the image needs to be corrected; and if the image does not include the photographing restriction element, the user device determines that the image does not need to be corrected.

Here, the relationship between the user of the user device and the recognized individual is the relationship between the photographer and the photographer, and it is determined whether the photographer and the photographer know each other, whether the photographer allows the photographer to treat him. / She is photographing or whether the photographer is a celebrity and is not restricted to being photographed by the photographer. When it is determined based on the relationship that the photograph of the photographer is restricted, it is determined that the image includes the photographing restriction element.

Meanwhile, if the identified object is a photography restriction object, the user device determines that the image includes a photography restriction element. The photography-restricted item may be a copyrighted item (such as a piece of art) or may be any item that is limited to photography by an ordinary individual for various reasons.

Whether the identified object is a photographic restriction object is determined by using any of a variety of methods. For example, the memory of the user device can store a database of photographic restricted objects, and the user device can compare the identified objects with the stored database. Alternatively, the external server can store a library of photographic restricted objects, and the user device can communicate with an external server to access the database and compare the identified objects to the stored database. Alternatively, the user device can transmit the image of the identified object to an external server, and then the external server can compare the transmitted image to the image stored in its database and transmit the result of the comparison to the user. Device. Alternatively, when the user enters a specific place (such as a museum), the user device can receive the database of the photography restriction object in the specific place, and compare the identified object with the database. . The determination of whether the identified object is a photographic restriction object can be performed via any of a variety of other methods.

Meanwhile, if the recognized position is the photographing restriction position, the user device determines that the image includes the photographing restriction element. The photography restriction position may be a location (such as a military security zone) that is photographed due to a security restriction or a location (such as a concert venue) that is photographed due to copyright restrictions. In addition, location can limit photography by unauthorized individuals for a variety of reasons.

Whether or not the identified position is a photography restriction is determined by using any of various methods. For example, the memory of the user device can store a database of photo-restricted locations, and the user device can compare the identified location with the stored database. Alternatively, the external server can store a database of photographic limited locations, and the user device can communicate with an external server to access the database and compare the identified locations to the stored database. Alternatively, the user device can determine whether the identified location is a forbidden photographic area by recognizing and analyzing a flag included in the image, such as a photographic area-inhibited flag.

If it is determined in operation 204 that the image includes a photographing restriction element, the user device may correct the image in operation 205. In particular, the user device can correct the image such that the photo-restricting elements included in the image are indistinguishable or unrecognizable. For example, the user device can perform mosaic processing or blurring processing on the photography-restricted person or object. Alternatively, the user device may limit the individual or object by arranging the photography to limit the image captured by the individual or object, or by presetting the alternate image. If the image is captured at the photographing limit position, the user device may perform mosaic processing or blurring processing to make the image indistinguishable or unrecognizable, or the image may be arranged by replacing the image by preset. In addition, the user device can correct the image such that the image-restricting elements included in the image are indistinguishable or unrecognizable. The method of correcting the image will be described in detail later.

If it is determined in operation 204 that the image does not include the photographing restriction element, the method ends without correcting the image.

Operations 202 through 205 of FIG. 2 are described as being performed by the user device, but alternatively some or all of operations 202 through 205 may be performed by a device or server coupled to the user device, as will be described in detail below.

3 through 6 are diagrams for describing a method of correcting an image including a photographing restriction element, according to an exemplary embodiment.

Referring to FIG. 3, the smart phone 100-2 of the user A captures an image including the user B and the user C. Subsequently, the smart phone 100-2 determines whether the image includes a photography restriction individual, and if the image includes a photography restriction individual, corrects the image.

In FIGS. 4 to 6, the smart phone 100-2 determines whether or not the image includes a photographing restriction person by using a different method, and adjusts the image.

Referring to FIG. 4, the smart phone 100-2 recognizes the faces of the user B and the user C by performing face recognition on the original image 400a. The smart phone 100-2 can determine whether the degree of intimacy between the user A and the user B and the user A and the user C is lower than a predetermined level. For this purpose, the smart phone 100-2 compares the recognized face with the photo contained in the address book stored in the smart phone 100-2. The address book stored in the smart phone 100-2 can store people's photos along with people's names and phone numbers.

In FIG. 4, as a result of comparing the recognized face with the photo included in the address book, the smart phone 100-2 determines that the user B's photo is in the address book. The smart phone 100-2 determines that the degree of intimacy between the user A and the user B is equal to or higher than the predetermined level and that the user B is an individual for photography.

Meanwhile, when the smart phone 100-2 cannot find the photo of the user C in the address book, the smart phone 100-2 determines that the degree of intimacy between the user A and the user C is lower than the predetermined level and the user C is a photography restriction individual. In other words, the smart phone 100-2 can determine that the user A and the user C are not related to each other. Therefore, the smart phone 100-2 determines that the original image 400a includes the photographing restriction element and corrects the original image 400a. For example, the smart phone 100-2 can correct a portion of the image corresponding to the user C to be unrecognizable or indistinguishable. In detail, as shown in FIG. 4, the smart phone 100-2 performs a mosaic process on the face area of the user C who is limited to photography and outputs the corrected image 400b.

In an exemplary embodiment, the mosaic process involves outputting a face area of the user C at a lower resolution. The lower resolution will be in a position so that the face area will be unrecognizable or unrecognizable.

The photographing restriction element is determined by comparing the recognized face with the photo included in the address book, because when the individual is registered in the address book, the individual is likely to know the user A of the smart phone 100-2. In other words, the smart phone 100-2 determines the relationship between the user A and the subject by using the address book, and determines whether or not the photographing restriction element is included in the original image 400a based on the relationship.

Referring to FIG. 5, the smart phone 100-2 recognizes the faces of the user B and the user C by performing face recognition on the original image 500a. The smart phone 100-2 can determine whether the degree of intimacy between the user A and the user B and the user A and the user C is lower than a predetermined level. For this purpose, the smart phone 100-2 compares the recognized face with the album stored in the smart phone 100-2.

In FIG. 5, the smart phone 100-2 determines that the user B is included in at least one of the photos by comparing the recognized face with the photo. The smart phone 100-2 determines that the degree of intimacy between the user A and the user B is equal to or higher than the predetermined level and that the user B is an individual for photography.

Meanwhile, when the smart phone 100-2 cannot find the photo of the user C in the address book, the smart phone 100-2 determines that the degree of intimacy between the user A and the user C is lower than the predetermined level and the user C is a photography restriction individual. In other words, the smart phone 100-2 can determine that the user A and the user C are not related to each other. Therefore, the smart phone 100-2 determines that the original image 500a includes the photographing restriction element and corrects the original image 500a. For example, the smart phone 100-2 can correct a portion of the image corresponding to the user C such that the user C is indistinguishable or unrecognizable. In detail, as shown in FIG. 5, the smart phone 100-2 performs a mosaic process on the face area of the user C who is limited to photography and outputs the corrected image 500b.

The photographing restriction element is determined by comparing the recognized face with the photo contained in the album, because when the individual is registered in the album, the individual is likely to know the user A of the smart phone 100-2. In other words, the smart phone 100-2 determines the relationship between the user A and the subject by using the photo of the album, and determines whether or not the photographing restriction element is included in the original image 500a based on the relationship.

Referring to FIG. 6, the smart phone 100-2 recognizes the faces of the user B and the user C by performing face recognition on the original image 600a. The smart phone 100-2 can determine whether the degree of intimacy between the user A and the user B and the user A and the user C is lower than a predetermined level. To this end, the smart phone 100-2 determines the relationship between the user A and the user B and the user C by analyzing the social networking service (SNS) usage record of the user A. The SNS in the exemplary embodiment may be a web-based or internet-based online SNS, and may be a platform that constitutes a social network or social relationship.

In FIG. 6, if the degree of intimacy between the user A and the user B is determined to be equal to or higher than the predetermined level by analyzing the SNS usage record, the smart phone 100-2 determines that the user B is an individual for photography. . For example, the degree of intimacy can be determined by whether the user A and the user B are friends on the SNS, whether one of the user A and the user B has written a post about the other person. Comments and more. Each of the factors may have the same weight or a different weight based on user settings or by default. For example, the weight "3" may indicate the fact that User A and User B are friends on the SNS, and the weight "0.2" may indicate that each of User A or User B posts on the other's post. A comment. When the predetermined level is set to "10", if User A and User B are friends (ie, weight "3") and they post 100 comments on the other's post (ie, weight "20" The degree of intimacy (i.e., "23") is judged to be higher than the predetermined level (i.e., "10").

Meanwhile, if the smart phone 100-2 determines that the degree of intimacy between the user A and the user C is lower than the predetermined level by analyzing the SNS usage record, the smart phone 100-2 determines that the user C is the photography restriction. personal. In other words, the smart phone 100-2 can determine that the user A and the user C are not related to each other. Therefore, the smart phone 100-2 determines that the original image 600a includes the photographing restriction element and corrects the original image 600a. For example, the smart phone 100-2 can correct a portion of the image corresponding to the user C such that the user C is indistinguishable or unrecognizable. In detail, as shown in FIG. 6, the smart phone 100-2 performs a mosaic process on the face area of the user C who is limited to photography and outputs the corrected image 600b.

The smart phone 100-2 can simultaneously analyze the address book, the album, and the SNS usage of determining the degree of intimacy between the user A and the user B and between the user A and the user C. For example, the smart phone 100-2 may consider the following factors determining the degree of intimacy between the user A and the user B: whether the user B is registered in the address book of the user A, and the plurality of users B The image is stored in the album of the user A, whether the user A and the user B are friends on the SNS, and whether one of the user A and the user B has written a comment on the other party's post. The exemplary embodiments of the present invention are not limited thereto, and other factors that determine the degree of intimacy between the photographer and the photographer may be considered.

7 through 12 are diagrams for describing a method of correcting an image including a photo restriction individual, according to an exemplary embodiment.

In Fig. 7, the blurring process is performed on the photographing restriction element. Referring to FIG. 7, the original image 700a includes a user B and a user C. Here, it is assumed that the user C is a photographing restriction element. Therefore, the area in which the original image 700a of the user C is displayed is blurred. In the corrected image 700b, the user C is blurred. Meanwhile, since the method of performing blurring processing in a specific region of an image is widely known, details thereof are not provided herein.

In Fig. 8, an alternative image of the photographic limiting element is arranged by the background image. Referring to FIG. 8, the original image 800a includes a user B and a user C. Here, it is assumed that the user C is a photographing restriction element. Therefore, the user C is arranged by the background image.

The user C uses the background image to delete the user C from the original image 800a and replaces the deleted user C area with an image similar to the background image surrounding the user C.

Referring to the corrected image 800b of FIG. 8, the area inside the edge 3-1 of the user C is filled with the same or similar pixel value as the pixel outside the edge 3-1. For example, the inner and adjacent pixels 802 of the edge 3-1 are configured to have the same or similar color, saturation, and brightness as the pixels 801 outside and adjacent the edge 3-1. Moreover, pixels adjacent to pixel 802 (from pixels inside edge 3-1) are set to have similar color, saturation, and brightness as pixel 802. Meanwhile, since any of various techniques commonly used in the prior art can be applied to a method of arranging a specific region of an image by a background image, details thereof are not provided herein.

In Figure 9, an alternate image of the photographic limiting element is arranged by pre-storing the background image. Referring to FIG. 9, the original image 900a includes a user B, a user C, and a structure 901. Here, it is assumed that the user C is a photographing restriction element. In the original image 900a, the user C and the structure 902 partially overlap each other. Therefore, the area of the user C deleted by the user C from the original image 900a and deleted is filled with the pre-stored image of the structure 902. Referring to the corrected image 900b, the area of the user C that overlaps the structure 902 is replaced by the pre-stored image of the structure 902. Here, all images of the structure 902 can be pre-stored in the user device or server that corrects the original image 900a.

In Fig. 10, an alternative to the photographing restriction element is shown in Fig. 10, and an alternative image of the photographing restriction element is arranged by pre-storing the substitute image 1001. Referring to FIG. 10, the original image 1000a includes a user 2 and a user 3. Here, it is assumed that the user 3 is a photographing restriction element. The corrected image 1000b is produced by replacing the user 3 with the pre-stored replacement image 1001. In FIG. 10, the pre-storage substitute image 1001 is selected to replace the face of the user C. Alternatively, however, the pre-stored alternate image 1001 may be an image used to replace the user C.

When the substitute image of the individual's face (ie, the substitute image of the photography limiting component) is generated by the substitute image, the substitute image may be edited or corrected based on the direction and angle of the face in the original image, which will be referred to below with reference to FIG. 11 and This is described in detail in FIG.

Referring to Figure 11, the face of the individual in the original image 1100a is facing forward. Thus, as shown in the corrected image 1100b, the face of the individual is arranged by facing the forward alternate image 1110 according to the direction and angle of the face of the individual in the original image 1100a.

Referring to FIG. 12, the face of the individual in the original image 1200a is facing the side. Thus, as shown in the corrected image 1200b, the individual's face is arranged by facing the lateral alternative image 1210 based on the direction and angle of the individual's face in the original image 1200a.

Similarly, the individual's face can be arranged by replacing the image according to the direction and angle of the individual's face, and thus a natural image can be obtained.

13 through 16 are flowcharts of a method of correcting an image including a photography-restricted individual, in accordance with an exemplary embodiment.

Referring to Figure 13, in operation 1301, the user device captures an image containing the individual. Here, the user device may be any of a general photographing device (such as a camera or a video camera), a wearable device (such as a smart eyeglass or a smart watch), and a mobile device (such as a smart phone or a tablet PC). Alternatively, it can be any other variety of devices that can be photographed.

In operation 1302, the user device performs facial recognition on an individual in the image. In particular, facial recognition can be performed by extracting features from the image detection face region and extracting features from the detected face regions.

In operation 1303, the user device determines the relationship between the photographer (ie, the user of the user device) and the subject (ie, the individual in the image) by using the data stored in the user device. . In other words, the user device determines whether the photographer and the photographer know each other.

In operation 1304, the user device determines whether the image contains an individual that is not related to the photographer. At this time, the method of determining the relationship between the photographer and the subject and determining whether the person not associated with the photographer is included in the image may vary, and some exemplary embodiments will be described below with reference to FIGS. 14 to 16.

If it is determined in operation 1304 that the individual not associated with the photographer is included in the image, then in operation 1305 the user device performs a protection process on the individual not associated with the photographer. In other words, the user device corrects the image so that the individual unrelated to the photographer is indistinguishable. The method of correcting an image has been described above with reference to FIGS. 7 to 12.

Referring to FIG. 14, in operation 1401, the smart phone captures an image containing a person.

In operation 1402, the smart phone performs facial recognition on an individual included in the image. In particular, facial recognition can be performed by extracting features from the image detection face region and extracting features from the detected face regions.

In operation 1403, the smart phone determines whether the individual is in the address book of the smart phone. In particular, the smart phone determines whether the individual matches at least one of the photos included in the address book. If it is determined that the individual is not in the address book, then operation 1404 is performed.

In operation 1404, the smart phone performs a mosaic process on the individual. Alternatively, the smart phone can perform blurring or background image processing on an individual.

Referring to FIG. 15, in operation 1501, the smart phone captures an image containing a person.

In operation 1502, the smart phone performs facial recognition on an individual included in the image. In particular, facial recognition can be performed by extracting features from the image detection face region and extracting features from the detected face regions.

In operation 1503, the smart phone determines whether the individual is included in the photo stored in the album of the smart phone. In particular, the smart phone determines whether the individual is included in at least one of the photos included in the album. If it is determined that the individual is not included in the at least one photo, then operation 1504 is performed.

In operation 1504, the smart phone performs a mosaic process on the individual. Alternatively, the smart phone can perform blurring or background image processing on an individual.

Referring to FIG. 16, in operation 1401, the smart phone captures an image containing a person.

In operation 1602, the smart phone performs facial recognition on an individual included in the image. In particular, facial recognition can be performed by extracting features from the image detection face region and extracting features from the detected face regions.

In operation 1603, the smart phone analyzes the SNS usage record of the photographer (i.e., the user of the smart phone). The smart phone can store various records of users who use smart phones and can analyze SNS usage records.

In operation 1604, the smart phone determines whether the individual is registered as a friend of the photographer on the SNS. If it is determined that the individual is not registered as a friend of the photographer on the SNS, then operation 1605 is performed.

In operation 1605, the smart phone performs a mosaic process on the individual. Alternatively, the smart phone can perform blurring or background image processing on an individual.

Meanwhile, the smart phone is used as a user device in the methods of FIGS. 14 to 16, but alternatively, any of various devices capable of capturing images and storing contacts can be used.

17 and 18 are diagrams for describing a method of correcting an image including a photographing restriction element by using at least two user devices, according to an exemplary embodiment. In particular, in the methods of Figures 17 and 18, photography and image processing are performed by different user devices.

Referring to FIG. 17, the user A can capture an image containing a person by using the smart glasses 100 that are wearable devices. Here, the image includes user B and user C. The smart glasses 100 transmit images to the smart phone 100-2, which is also the device of the user A. After receiving the video, the smart phone 100-2 determines whether the image includes the photographing restriction element, and if it is determined that the photographing restriction element is included in the image, the smart phone 100-2 corrects the image.

Referring to FIG. 18, the smart phone 100-2 performs face recognition on a person included in the original image 1800a received from the smart glasses 100. After recognizing the faces of the user B and the user C included in the original image 1800a, the smart phone 100-2 compares the recognized faces with the photos stored in the address book in the smart phone 100-2. The address book stored in the smart phone 100-2 can store people's photos as well as people's names and phone numbers.

In FIG. 18, if it is determined that the photo of the user B is in the address book by comparing the recognized face with the photo in the address book, the smart phone 100-2 determines that the user B is an individual for photography. Meanwhile, when the photo of the user C is not found in the address book, the smart phone 100-2 determines that the user C is a photography restriction individual. Therefore, the smart phone 100-2 determines that the image includes the photographing restriction element and corrects the image. For example, the smart phone 100-2 can output the corrected image 1800b by the user C programmed to limit the individual using the background image. The method of arranging the user C by the background image has been described in detail above with reference to FIG.

Alternatively, the smart phone 100-2 can compare the face identified from the original image 1800a with the photo stored in the album in the smart phone 100-2 or by analyzing the SNS usage record of the user A. To determine that photography limits individuals. The method has been described above with reference to FIGS. 5 and 6.

In addition, in FIG. 18, the smart phone 100-2 arranges the user C by the background image, but alternatively, the user can be arranged by performing mosaic processing or blurring processing on the C user 3 or by using an alternative image. C to correct the original image 1800a as described above with reference to Figures 7-12.

19 is a flow diagram of a method of modifying an image containing a photographic limiting element by using at least two user devices, in accordance with an illustrative embodiment.

Referring to FIG. 19, in operation 1901, the wearable device captures an image containing a person. Here, the wearable device (smart glasses or smart watch) can have a photographing function.

In operation 1902, the wearable device transmits the image to the smart phone. Here, the image is transmitted to a smart phone owned by a user of the wearable device, and a device such as a tablet PC can be used instead of the smart phone.

In operation 1903, the smart phone performs face recognition on an individual included in the image. In particular, facial recognition is performed by extracting features from the image detection face region and extracting features from the detected face regions.

In operation 1904, the smart phone determines if the individual is in the address book of the smart phone. In other words, the smart phone compares the individual with the photos stored in the address book to determine if the individual is included in at least one of the photos.

If it is determined that the individual is not in the address book, then in operation 1905 the individual substitute image is arranged by the background image.

20 to 22 are diagrams for describing a method of correcting an image including a photographing restriction element via communication with a device of a subject, according to an exemplary embodiment. In detail, according to the method of Figs. 20 to 22, the apparatus of the photographer transmits a response of whether or not the photographing is permitted to the apparatus of the photographer.

Referring to FIG. 20, user A's smart phone 100-2 captures an image containing user B. When the image is captured, the smart phone 100-2 transmits the photography notification to the smart phone 200 of the user B who is the photographer. Here, the photography notification may include identification (identification; ID) information identifying the user A.

After receiving the photographing notification, the smart phone 200 determines that the photographer is the user A by using the ID information included in the photographing notification, and determines whether or not the photographer (user A) is allowed to capture the image of the user B. A method of discriminating a photographer by using ID information included in a photographing notification and determining whether or not photographing is permitted will be described in detail below with reference to FIG.

After determining whether or not photography is permitted, the smart phone 200 transmits a response to the smart phone 100-2. The smart phone 100-2 stores the image based on the response to the corrected image or without correction.

In FIG. 21, the smart phone 100-2 of the user A transmits a photographing notification to the smart phone 200 of the user B, and the smart phone 200 determines whether or not photography is permitted and transmits the response to the smart phone 100-2.

Referring to FIG. 21, the smart phone 100-2 transmits a photographing notification including the ID information of the user A to the smart phone 200 while photographing the user B. Here, the ID information included in the photography notification may include the name and photo of the user A.

The smart phone 200 determines whether or not the user A is allowed to take a picture based on the ID information of the user A included in the photographing notification received from the smart phone 100-2. In detail, the smart phone 200 compares the photo of the user A included in the ID information with the photo stored in the address book, or stores the name of the user A included in the ID information in the address book. The names are compared. If it is determined that the photo or name of the user A is not in the address book, the smart phone 200 does not allow photography. On the other hand, if the photo or name of the user A is in the address book, the smart phone 200 allows photography.

Alternatively, the smart phone 200 can compare the photo of the user A included in the ID information with the photo stored in the album of the smart phone 200 or by analyzing the SNS usage record of the user B. Determine if photography is allowed.

In FIG. 21, if the photo of the user A is not in the address book of the smart phone 200, the smart phone 200 determines that photography is not allowed and transmits the photography prohibition message to the smart phone 100-2. At this time, the smart phone 200 transmits the photo of the user B together with the photography prohibition message to the smart phone 100-2.

Subsequently, referring to FIG. 22, the smart phone 100-2 performs face recognition on the original image 1600a. After performing face recognition, the smart phone 100-2 compares the photo of the user B received from the smart phone 200 with the photography prohibition message with the face recognized in the original image 1600a. If the recognized face matches the photo of the user B, the smart phone 100-2 performs a mosaic process on the face area of the user B in the original image 1600a and outputs the corrected image 1600b. Alternatively, the smart phone 100-2 can be modified by performing blurring processing on the user B, by arranging the substitute image of the user B with the background image, or by arranging the substitute image and the substitute image of the user B. Original image 1600a.

23 and 24 are flowcharts of a method of correcting an image including a photographing restriction element via communication with a device of a subject, in accordance with an exemplary embodiment.

Referring to FIG. 23, in operation 2301, the photographer's device transmits a photographing notification including the photographer's ID information to the subject's device. For example, a photographer's device may search for devices located within a particular range during photography and transmit a photography notification to the discovered device. The device can be searched for and transmitted by using near field communication (such as Wi-Fi direct or Bluetooth), and it may be necessary to approve the device of the photographer for transmitting the photographic notification.

In operation 2302, the device of the photographer determines whether or not photography is permitted based on the ID information of the photographer included in the photography notification.

If it is determined in operation 2303 that photography is permitted, then operation 2307 is performed. In operation 2307, the photographer's device transmits a photography permission message to the photographer's device. Subsequently, the photographer's device determines that the captured image does not contain the photographing restriction element and stores the captured image without correction.

If it is determined in operation 2303 that photography is not allowed, then operation 2304 is performed. In operation 2304, the photographer's device transmits the photography prohibition message along with the photographer's photo to the photographer's device.

In operation 2305, the photographer's device performs facial recognition on the people included in the captured image.

In operation 2306, the photographer's device compares the face recognized from the captured image with the photo of the subject, and performs a protection process on the face matching the photographer's photo. In other words, the mosaic processing, the blurring processing, the layout processing by the background image, or the arrangement processing by replacing the image is performed on the face matching the photo of the photographer.

Fig. 24 is a flow chart showing a method of determining whether or not photography is permitted by using an address book of a device of a photographer.

Referring to FIG. 24, in operation 2401, the photographer's smart phone transmits a photographing notification including the photographer's ID information to the photographer's smart phone.

In operation 2402, the photographer's smart phone determines whether the photographer is in the address book by using the photographer's ID information. In detail, the name or photo of the photographer included in the ID information is compared with the name or photo stored in the address book. Alternatively, the photographer's smart phone can be determined by comparing the photographer's ID information with a photo album stored in the photographer's smart phone, or by analyzing the photographer's SNS usage record. Whether to allow photography.

If it is determined in operation 2402 that the photographer is in the address book, then in operation 2406, a smart phone that allows photography and that is transmitted by the photographer's smart phone to the photographer is determined.

If it is determined in operation 2402 that the photographer is not in the address book, it is determined that photography is not allowed and operation 2403 is performed.

In operation 2403, the photographer's smart phone transmits the photography prohibition message along with the photographer's photo to the photographer's smart phone.

In operation 2404, the photographer's smart phone performs facial recognition on the people included in the captured image.

In operation 2405, the photographer's smart phone compares the recognized face with the photo of the photographer and performs a protection process on the face matching the photo of the photographer. In other words, the mosaic processing, the blurring processing, the layout processing by the background image, or the arrangement processing by replacing the image is performed on the face matching the photo of the photographer.

25 and 26 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server, according to an exemplary embodiment.

Referring to FIG. 25, the smart phone 100-2 of the user A captures the original image including the user B and the user C, and transmits the original image to the server 300 for image processing. The server 300 determines whether or not the photographing restriction element is included in the original image, and if it is determined that the photographing restriction element is included, corrects the original image and transmits the corrected image to the smart phone 100-2 of the user A. A method of determining whether or not the photographing restriction element is included in the image by the server 300 will now be described in detail with reference to FIG.

Referring to FIG. 26, the server 300 performs face recognition on the original image 2600a received from the smart phone 100-2 of the user A to recognize the faces of the user B and the user C. The server 300 determines whether the user B and the user C have allowed the user A to take a picture of himself by referring to the privacy protection information 2610 pre-stored in the server 300.

Here, the privacy protection information 2610 defines, for at least one user, another user who is allowed to take a photo. Referring to FIG. 26, another user who is allowed to take a picture is mapped according to the user in the privacy protection information 2610. For example, User A does not allow anyone to photograph himself/herself. Further, User B only allows User A to photograph himself/herself, and User C allows User B and User D to photograph himself/herself.

The server 300 compares the user B and the user C identified from the original image 2600a with the privacy protection information 2610 to determine whether or not photography is permitted. According to the privacy protection information 2610, since the user B allows the user A to take an image of himself/herself and the user C does not allow it, the server 300 determines that the user C is a photography restriction element.

Meanwhile, in order to determine whether or not photography is permitted, the server 300 needs to check whether the photographer of the original image 2600a is the user A. Therefore, the smart phone 100-2 of the user A can transmit the ID information of the user A to the server 300 together with the original image 2600a.

After determining that the user C is the photographing restriction element, the server 300 generates the corrected image 2600b by performing mosaic processing on the user C in the original image 2600a, and transmits the corrected image 2600b to the wisdom of the user A. Type telephone 100-2. Alternatively, the server 300 may correct the original image 2600a by performing blurring processing, formatting processing by background images, or by replacing the image processing.

FIG. 27 is a flowchart of a method of correcting an image including a photographing restriction element by using a server, according to an exemplary embodiment.

Referring to Figure 27, in operation 2701, the user device captures an image.

In operation 2702, the user device transmits the image to the server and requests the server to perform image processing.

In operation 2703, the server performs facial recognition on an individual included in the image received from the user device.

In operation 2704, the server determines whether the identified individual is allowed to be photographed by referring to the pre-stored privacy protection information. In other words, the server determines whether the identified individual is allowed to be photographed by the photographer of the image. A method of determining whether or not photography is permitted by using privacy protection information has been described above with reference to FIG.

If it is determined in operation 2705 that the identified individual is not allowed to be photographed, then in operation 2706 the server performs a protection process on the identified individual. In operation 2707, the server transmits the corrected image to the user device.

If it is determined in operation 2705 that the identified individual is allowed to be photographed, then in operation 2707 the server transmits the image to the user device without correction.

28 to 31 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server, according to other exemplary embodiments. In detail, according to the method of FIGS. 28 to 31, before transmitting the original image to the server, the user device performing the photographing encodes the original image by using the encryption key managed by the server, and thus the photographer cannot Access the original image processed by the server.

Referring to FIG. 28, the smart phone 100-2 of the user A receives the encryption key from the server 300. Here, the encryption key is managed by the server 300. The smart phone 100-2 of the user A captures the image including the user B and the user C, and encodes the image by using the encryption key received from the server 300. Here, since the smart phone 100-2 of the user A encodes the image immediately after capturing the image, the user A cannot access the image.

As shown in FIG. 29, the smart phone 100-2 of the user A transmits the encoded image to the server 300.

Referring to Figure 30, server 300 decodes the encoded image. Since the encryption key used to encode the encoded image is managed by the server 300, the server 300 can decode the encoded image using the encryption key. The server 300 recognizes the faces of the user B and the user C by performing face recognition on the decoded image. After performing facial recognition, the server 300 determines whether the user B and the user C are allowed to be photographed by using the privacy protection information 3010 pre-stored in the server 300. A method of determining whether or not photography is permitted by using privacy protection information has been described above with reference to FIG.

If it is determined based on the privacy protection information 3010 that the photographing user C is not allowed, the server 300 can correct the decoded image by performing a mosaic process on the user C.

Subsequently, the server 300 transmits the corrected image to the smart phone 100-2 of the user A as shown in FIG.

32 is a flowchart of a method of correcting an image including a photographing restriction element by using a server, according to another exemplary embodiment.

Referring to Figure 32, in operation 3201, the user device captures an image containing the individual.

In operation 3202, the user device encodes the image by using an encryption key received from the server. Here, the encryption key is managed by the server.

In operation 3203, the user device transmits the encoded image to the server.

In operation 3204, the server decodes the encoded image received from the user device and performs facial recognition to identify the individual included in the decoded image.

In operation 3205, the server determines whether the identified individual is allowed to be photographed by using privacy protection information pre-stored in the server. A method of determining photography by referring to privacy protection information has been described above with reference to FIG.

In operation 3206, the server performs a protection process on an individual who is not allowed to be photographed. For example, the server may perform mosaic processing, blurring processing, orchestration processing by background images, or by arrangement processing of substitute images on an individual who is not allowed to be photographed.

In operation 3207, the server transmits the corrected image to the user device.

33 to 37 are diagrams for describing a method of correcting an image including a photographing restriction object, according to an exemplary embodiment.

Referring to FIG. 33, user A's smart phone 100-2 captures an image containing user B and statue 20. The smart phone 100-2 of the user A transmits the image transmission server 300 to request the server 300 to perform image processing. The server 300 determines whether the image includes the photographing restriction element, and if the photographing restriction element is included, corrects the image and transmits the corrected image to the smart phone 100-2 of the user A. A method of determining whether or not the photographing restriction element is included in the image and correcting the image by the server 300 will be described below with reference to FIGS. 34 to 37.

Referring to FIG. 34, the server 300 recognizes an object included in the original image 3400a. In other words, the statue 20 is identified by extracting features (such as shape and color) of the statue 20 contained in the original image 3400a. The server 300 determines whether the statue is stored in the photographic restricted object database stored in the server 300. The Photography Restricted Objects Library stores photos of photography-restricted objects.

If the statue 20 is in the photography restricted object database, the server 300 determines that the statue 20 is a photographing restriction element. The server 300 performs a mosaic process on the statue 20 in the original image 3400a to generate a corrected image 3400b, and transmits the corrected image 3400b to the user A's smart phone 100-2.

Meanwhile, in FIGS. 35 to 37, the image is corrected by using different methods. A method of recognizing an object in an image and determining whether the photographing restriction element is included in the image by using the pre-stored photographing restriction object database is the same as the method described with reference to FIG. However, in Fig. 35, when the image is corrected, the corrected image 3500b is generated by performing blurring processing. In FIG. 36, the corrected image 3600b is generated by performing an orchestration process by the background image as described above with reference to FIG. In FIG. 37, the corrected image 3700b is generated by arranging the substitute image 3710.

38 is a flow chart of a method of modifying an image containing a photographic restriction object, in accordance with an exemplary embodiment.

Referring to Figure 38, in operation 3801, the user device captures an image.

In operation 3802, the user device transmits the image to the server.

In operation 3803, the server identifies the object contained in the image. In particular, the objects contained in the image are detected and the features (such as shape and color) of the detected object are extracted.

In operation 3804, the server determines whether the identified object is included in a library of photographic restricted objects pre-stored in the server. The Photography Restricted Objects Database stores a list of photography restricted objects.

If it is determined in operation 3804 that the identified object is in the photography restricted object database, then in operation 3805 the server corrects the image by performing a protection process on the identified object, and the corrected image is performed in operation 3806. Transfer to the user device.

If it is determined in operation 3804 that the identified object is not in the photographic restricted object database, then in operation 3806 the server transmits the image to the user device without correction.

39 to 41 are diagrams for describing a method of correcting an image captured at a photographing restriction position, according to an exemplary embodiment.

Referring to FIG. 39, the smart phone 100-2 of the user A captures an image of the stadium to be photographed, and transmits the image to the server 300 to request the server 300 to perform image processing. At this time, the smart phone 100-2 transmits the location information together with the video to the server 300. The location information is information about a location at which an image is captured by the smart phone 100-2, wherein the smart phone 100-2 determines the location by using the GPS embedded therein and transmits the determined location to the server 300. Alternatively, the smart phone 100-2 can generate a location via any of a variety of methods (eg, by using a base station or access point (AP) connected to the smart phone 100-2 to determine the location) News.

The server 300 determines whether the image contains the photographing restriction element by using the position information. A method of determining whether an image contains a photographing restriction element by the server 300 will be described in detail below with reference to FIGS. 40 and 41.

If it is determined that the image includes a photographing restriction element, the server 300 corrects the image and transmits the corrected image to the smart phone 100-2.

Referring to FIG. 40, the server 300 determines the position of the captured image by analyzing the position information received from the smart phone 100-2, and determines whether the determined position is the shooting restriction position. In other words, the server 300 determines whether the determined position is pre-stored in the photographing limit position database of the server 300. Here, the photography limit location database can store information about locations that limit photography by unauthorized individuals, such as concert venues, museums, military security zones. When the server 300 determines that the determined location is a location stored in the database, the server 300 can determine whether the repository stores the name of the location that exactly matches the name of the determined location. Alternatively, the server 300 may evaluate the similarity between the determined location and the location stored in the repository, and determine whether the determined location is in the repository based on the degree of similarity.

When it is determined that the determined position is in the photographing limit position database, the server 300 determines that the image includes the photographing restriction element and corrects the image. In FIG. 40, the server 300 performs a mosaic process on the original image 4000a to produce a corrected image 4000b.

Referring to Fig. 41, the server 300 determines the position by analyzing the position information received from the smart phone 100-2, and determines whether the position is the shooting restriction position. In other words, the server 300 determines whether the determined position is pre-stored in the photographing limit position database of the server 300. Here, the photography limit location database can store information about locations that limit photography by unauthorized individuals, such as concert venues, museums, military security zones.

When it is determined that the determined position is in the photographing limit position database, the server 300 determines that the image includes the photographing restriction element and corrects the image. In FIG. 41, the server 300 generates the corrected image 4100b by using the substitute image to arrange the original image 4100a. Alternatively, the server 300 may correct the original image 4100a by performing blurring processing on the original image 4100a.

Meanwhile, the server 300 corrects the images in FIGS. 39 to 41, but alternatively, the server 300 can determine whether the determined position is the photographing limit position by analyzing the position information received from the smart phone 100-2, and The result of the determination is transmitted to the smart phone 100-2. The smart phone 100-2 can then correct the image based on the received results.

FIG. 42 is a flowchart of a method of correcting an image captured at a photographing restriction position, according to an exemplary embodiment.

Referring to FIG. 42, in operation 4201, the user device captures an image. Here, the user device may be a general photographic device (such as a camera or a video camera), a wearable device (such as a smart glasses or a smart watch) or a mobile device (such as a smart phone or a tablet PC), or Any other device that can be photographed.

In operation 4202, the user device determines the location of the captured image and transmits the location information to the server. Here, the user device can determine the location by using the GPS embedded therein. Alternatively, the user device can determine the location by using the location of the base station or AP connected to the user device.

In operation 4203, the server determines whether the determined position is a photographing restriction position. In detail, the server determines the location by analyzing the received location information, and compares the determined location with a library of camera location locations pre-stored in the server. Here, the photography limit location database can store information about locations that limit photography by unauthorized individuals, such as concert venues, museums, military security zones.

If the determined position is in the photographing limit position database, the server determines that the image contains the photographing restriction element and performs operation 4204. In operation 4204, the server corrects the image and transmits the corrected image to the user device. In other words, the server generates the corrected image by performing mosaic processing, blurring processing on the image, or by replacing the image processing, and transmitting the corrected image to the user device.

43 to 45 are diagrams for describing a method of correcting an image captured at a photographing restriction position, according to other exemplary embodiments. In detail, in FIGS. 43 to 45, the user device determines that the position of the captured image is the photographing restriction position by recognizing the prohibition photographing flag.

Referring to FIG. 43, the smart phone 100-2 of the user A captures an image including the prohibition photography mark 30 and the user B. Here, the prohibition photography mark 30 may include symbols and phrases indicating that photography is prohibited as shown in FIG. 43, or may be in any other form. For example, the banned photographic sign 30 may include a bar code indicating a photography prohibition, a quick response (QR) code, or a color code. The smart phone 100-2 prevents the photographer from inadvertently or intentionally capturing the photo when the photo contains a bar code, QR code or color code that the photographer may not understand but indicates that photography is prohibited.

Referring to FIG. 44, the smart phone 100-2 of the user A recognizes the photo-inhibited flag 30 included in the original image 4400a. In detail, the smart phone 100-2 recognizes at least one of the symbol 30-1 and the phrase 30-2 included in the prohibition photography mark 30. After recognizing the prohibition photography mark 30, the smart phone 100-2 determines that the image is captured at the photographing restriction position. In other words, the smart phone 100-2 determines that the image contains a photographing restriction element. Therefore, the smart phone 100-2 performs a mosaic process on the original image 4400a to generate a corrected image 4400b.

Referring to FIG. 45, the smart phone 100-2 of the user A recognizes the photo-inhibited flag 30 included in the original image 4500a. In detail, the smart phone 100-2 recognizes at least one of the symbol 30-1 and the phrase 30-2 included in the prohibition photography mark 30. After recognizing the prohibition photography mark 30, the smart phone 100-2 determines that the image is captured at the photographing restriction position. In other words, the smart phone 100-2 determines that the image contains a photographing restriction element. Therefore, the smart phone 100-2 generates the corrected image 4500b by using the substitute image to arrange the original image 4500a.

FIG. 46 is a flowchart of a method of correcting an image captured at a photographing restriction position, according to another exemplary embodiment.

Referring to Figure 46, in operation 4601, the user device captures an image.

In operation 4602, the user device identifies the object contained in the image.

In operation 4603, the user device determines whether the identified object contains a photo-inhibited flag. If it is determined that the identified object includes a prohibited photo indicia, then in operation 4604 the user device corrects the image. Here, the user device can correct the image by performing a mosaic process, a blur process, or an arrangement process instead of the image on the image.

47 to 49 are diagrams for describing an example of restricting photography of an individual according to privacy protection information, according to an exemplary embodiment.

Referring to FIG. 47, user A's smart phone 100-2 captures an image including user B and user C. Subsequently, the smart phone 100-2 transmits the image to the server 300 and requests the server 300 to perform image processing. At this time, the smart phone 100-2 can encode the image by using the encryption key received from the server 300, and transmit the encoded image to the server 300.

The server 300 determines whether the image includes a photographing restriction element by using the privacy protection information 4710 pre-stored in the server 300. When the image is encoded, the server 300 decodes the encoded image by using the encryption key before determining whether the photography restriction element is included in the image. Here, the photography restriction time and location are mapped according to the user in the privacy protection information 4710.

A method of determining whether an image contains a photographing restriction element by using privacy protection information will be described in detail later with reference to FIGS. 48 and 49.

If it is determined that the image includes the photographing restriction element, the server 300 corrects the image and transmits the corrected image to the smart phone 100-2.

A method of determining whether an image includes a photographing restriction element and correcting an image by using the privacy protection information 4710 will be described in detail below with reference to FIGS. 48 and 49, and the method is performed by the server 300.

Referring to FIG. 48, the smart phone 100-2 transmits the encoded image to the server 300. Here, the smart phone 100-2 also transmits location information about the location of the captured image along with the encoded image to the server 300. The smart phone 100-2 determines the location by using the GPS embedded therein or the location of the base station or AP connected thereto, and can transmit the location information to the server 300. In FIG. 48, the smart phone 100-2 transmits "school" as location information to the server 300.

Referring to FIG. 49, the server 300 decodes the encoded image and performs face recognition on the decoded image to identify the faces of the user B and the user C. The server 300 determines the photography restriction time and position of the recognized face based on the privacy protection information 4910. The privacy protection information 4910 is based on at least one of the user storing the photo restriction time and location.

Based on the privacy protection information 4910 of FIG. 49, the user A does not allow photography from 24:00 to 06:00 and does not allow home photography. User B does not allow photography from 22:00 to 09:00 and does not set the photography limit position. User C does not set the shooting position and time and does not allow photography at school.

The server 300 determines whether or not the image includes the photographing restriction element based on the position information (school) and the capture time of the smart phone 100-2 from the user A. Therefore, the server 300 can receive the capture time of the captured image while receiving the image from the smart phone 100-2.

In FIG. 49, the server 300 determines the position time of the captured image as "school" and "21:00" based on the position information and the capture time received from the smart phone 100-2.

The server 300 determines that the photography of the user B is restricted from 22:00 to 09:00 by referring to the privacy protection information 4910, but the position is not limited. Since the capture time "21:00" determined by the server 300 is not within the photographing time limit of the user B, it is determined that the photographing by the user B is not restricted. In other words, the server 300 determines that the user B is not a photographing restriction element.

At the same time, the server 300 determines that the photography of the user C is restricted in the "school" by referring to the privacy protection information 4910, but the time is not limited. Since the position information ("school") determined by the server 300 is within the photographing restriction position of the user C, it is determined that the photographing of the user C is restricted. In other words, the server 300 determines that the user C is a photographing restriction element.

Therefore, the server 300 corrects the image by performing mosaic processing only on the user C.

FIG. 50 is a flowchart of a method of restricting photography of an individual based on privacy protection information, according to an exemplary embodiment.

Referring to FIG. 50, in operation 5001, the user device captures an image containing a person. Here, the user device may be any of a general photographing device (such as a camera or a video camera), a wearable device (such as a smart eyeglass or a smart watch), and a mobile device (such as a smart phone or a tablet PC). Alternatively, it can be any of a variety of devices that can be photographed.

In operation 5002, the user device encodes the image by using an encryption key received from the server. Here, the encryption key is managed by the server.

In operation 5003, the user device transmits the encoded image and the location information of the user device to the server.

In operation 5004, the server decodes the encoded image and performs facial recognition on the individual included in the decoded image. In particular, facial recognition can be performed by detecting a face region from the decoded image and extracting features from the detected face region.

In operation 5005, the server determines whether the individual is allowed to be photographed by using privacy protection information pre-stored in the server, based on the location information, and the capture time of the captured image received from the user device.

In operation 5006, the server performs a protection process on an individual who is not allowed to be photographed. In other words, the server can correct the decoded image by performing at least one of a mosaic process, a blur process, a background image layout process, and an alternate image layout process, so that the person who is not allowed to be photographed is indistinguishable. Or unrecognizable.

In operation 5007, the server transmits the corrected image to the user device.

FIG. 51 is a block diagram of a user device 1000, in accordance with an exemplary embodiment. Referring to FIG. 51, a user device 1000 in accordance with an exemplary embodiment may include a user interface 1100, a camera 1200, a memory 1300, a communication interface 1400, and a controller 1500. Additionally, controller 1500 can include image analyzer 1510 image modifier 1520.

Meanwhile, as described above, the user device 1000 according to an exemplary embodiment may be any of various devices capable of photographing, such as smart glasses, smart watches, smart phones, tablet PCs, cameras, or video recordings. Camera.

The user interface 1100 receives input from the user and displays the screen to the user, and can be configured differently depending on the type of user device 1000. For example, when the user device 1000 is a smart phone, the user interface 1100 can include a touch screen.

The camera 1200 may include a camera module, a smart watch, a smart phone, or a tablet PC embedded in the smart glasses. Camera 1200 can capture images (such as still images or moving images) and transmit the images to controller 1500.

The storage 1300 can include a memory that stores various segments of images or materials. For example, the storage 1300 can store an address book, a photo, a moving image, or a usage record of the user device 1000, or can store an image corrected by the controller 1500.

The communication interface 1400 can perform data communication with an external device or server via a wire or wirelessly.

The controller 1500 controls the operations of the components included in the user device 1000. In detail, the controller 1500 determines whether the image includes the photographing restriction element by analyzing the image, and corrects the image if it is determined that the image includes the photographing restriction element.

The image analyzer 1510 can identify the location of the person included in the image, the object included in the image, or the location of the captured image by analyzing the image, and based on the relationship between the user of the user device 1000 and the identified individual, At least one of the object and the identified location determines if the image needs to be corrected. In detail, the image analyzer 1510 can determine whether the user and the identified individuals are related to each other by using the data stored in the storage 1300, and if it is determined that the user and the identified individuals are not related to each other, determining the image needs Was corrected.

For example, the image analyzer 1510 can determine whether the identified individual is in the address book stored in the storage 1300, and if the identified individual is not in the address book, it can be determined that the user and the identified individual are not in each other. Related. The image analyzer 1510 can perform content-based image capture based on the similarity of the image content to determine whether the identified individual is in the address book.

Alternatively, the image analyzer 1510 may determine whether the identified individual is included in at least one photo stored in the storage 1300, and if the identified individual is not included in the at least one photo, the user and the identified Individuals are not related to each other.

Alternatively, the image analyzer 1510 can determine whether the user and the identified individuals are related to each other by analyzing the user's SNS usage record using the data stored in the storage 1300.

Alternatively, the image 1510 can determine whether the identified object or the recognized position limits the photographing by accessing the database of the external server via the communication interface 1400, and if it is determined that the identified object or the recognized position limit is determined For photography, the image analyzer 1510 can determine that the image needs to be corrected.

Alternatively, if the photographic mark is prohibited from being recognized in the image, the image analyzer 1510 can determine that the image needs to be corrected by determining that the recognized position is the photographic limit position.

Meanwhile, when the image analyzer 1510 determines that the image needs to be corrected, the image corrector 1520 corrects the image. The image corrector 1520 may perform at least one of a mosaic process, a blur process, a layout process by the background image, and an arrangement process by replacing the image on the recognized person, the recognized object, or the image. A method of correcting an image by the image corrector 1520 has been described above with reference to FIGS. 7 through 12.

At the same time, the components of the user device 1000 may perform operations for performing the methods described above with reference to FIGS. 1 through 50, even if not described with reference to FIG.

At the same time, the time point at which the captured image is corrected can be set differently. For example, the captured image can be corrected before being stored in the user device. Alternatively, the captured image can be stored in the user device and then corrected as it is transmitted to an external server or another user device. Alternatively, the captured image may not be stored in the user device that captured the captured image, but may be corrected and transmitted to an external server or another user device. Hereinafter, different time points of correcting the captured images according to an exemplary embodiment will be described.

52 is a diagram of an example of a method for correcting a captured image, modifying the captured image before storing the captured image in a user device, in accordance with an exemplary embodiment.

In FIG. 52, the smart phone 100-2 on the left side displays a plurality of images on the screen when the library application is executed, and when the user touches and selects the image 5210 from the plurality of images, the image 5210 is displayed as the right side. The magnified image 5220 shown in the smart phone 100-2.

The image 5210 selected from the plurality of images includes a photography restriction individual, and thus the mosaic processing is performed on the region 5211 corresponding to the face of the individual who restricted the photography. The mosaic processing is also performed on the area 5221 of the enlarged image 5220, which corresponds to the face of the individual who limits the photography.

The image is captured by the smart phone 100-2, and before the captured image is stored in the smart phone 100-2, it is determined whether the captured image includes a photographing restriction element. When the captured image includes a photographing restriction element, the captured image is corrected and then stored in the smart phone 100-2. In other words, the original image of the captured image is not stored in the smart phone 100-2, and thus only the corrected captured image can be viewed via the smart phone 100-2. Therefore, according to the current embodiment, the user of the smart phone 100-2 can only access the corrected captured image.

The method of determining whether an image includes a photographing restriction element and correcting an image has been described in detail above.

53 is a flowchart for explaining an example of correcting a captured image before storing the captured image in a user device, in accordance with an exemplary embodiment.

Referring to FIG. 53, in operation 5301, the user device captures an image. Here, the user device can be any device capable of photographing, such as a smart phone, a video camera or a tablet PC.

In operation 5302, the user device identifies at least one of an individual included in the image, an object included in the image, and a location at which the image is captured. A method of identifying a person, object or location has been described above with reference to FIG.

In operation 5303, the user device determines whether the image includes a photographing restriction element based on at least one of the identified relationship between the individual and the user of the user device, the identified object, and the recognized location. A method of determining whether an image contains a photographing restriction element has been described above with reference to FIG.

When it is determined in operation 5304 that the image includes a photographing restriction element, the user device corrects the image and stores the corrected image in operation 5305. When it is determined in operation 5304 that the image does not include the photographing restriction element, the user device stores the image without correction in operation 5306.

According to another exemplary embodiment, operations 5302 and 5303 in FIG. 53 may be omitted, and in operation 5304, the user device may determine that the image includes a photography limiting component based on a user input for selecting a particular object included in the image. . For example, when the user touches and remains touching a portion of the screen displaying the object, the user device can determine that the object is a photography limiting element.

54 and 55 are diagrams for explaining a method for correcting a captured image, and modifying an image stored in the user device while externally transmitting the image, according to an exemplary embodiment.

In FIG. 54, when the gallery application is executed, a plurality of images are displayed on the screen of the smart phone 100-2 on the left side, and when the image 5410 is touched from a plurality of images, the image 5410 is smart on the right side. The enlarged image 5420 is displayed on the telephone 100-2.

Although the image 5410 contains a photography-restricted individual, the mosaic processing is not performed on the area 5411 of the face corresponding to the individual who restricted the photography. Further, the mosaic processing is not performed on the area 5421 of the enlarged image 5420, and the area 5221 corresponds to the face of the photographing restriction individual.

In the current embodiment, since the smart phone 100-2 stores the image without correcting the image, the user can access the original image via the library application of the smart phone 100-2.

However, in the current embodiment, if the image stored in the smart phone 100-2 is to be transmitted to an external server or another user device, it is determined whether the image includes a photographing restriction element; and when the image includes a photographing restriction element The image is corrected and then transmitted to an external server or another user device.

Referring to FIG. 55, when the user selects the image 5510 from the plurality of images displayed on the smart phone 100-2 via the gallery application and selects the "send" button 5530 to share the image 5510, the smart phone 100-2 determines the image. Whether or not the 5510 includes a photographing restriction element, and when the image 5510 includes a photographing restriction element, the image 5510 is corrected before the image 5510 is shared.

In FIG. 55, the smart phone 100-2 transmits the image 5520 corrected by performing the mosaic processing in the area 5521 corresponding to the face of the photographing restriction person to the server 300 designated as the target.

FIG. 56 is a flowchart for explaining an example of correcting a captured image, and correcting an image stored in the user device while externally transmitting the image, according to an exemplary embodiment.

Referring to Figure 56, in operation 5601, the user device captures an image. Here, the user device can be any device capable of photographing, such as a smart phone, a video camera or a tablet PC.

In operation 5602, the user device stores the image in a memory included in the user device.

In operation 5603, the user device receives a request to transfer the image from the user to the external server or another user device.

In operation 5604, the user device identifies at least one of an individual included in the image, an object included in the image, or a location to capture the image. A method of identifying a person, object or location has been described above with reference to FIG.

In operation 5605, the user device determines whether the image includes a photographing restriction element based on at least one of the identified relationship between the individual and the user of the user device, the identified object, and the identified location. A method of determining whether an image contains a photographing restriction element has been described above with reference to FIG.

When it is determined in operation 5606 that the image includes a photographing restriction element, in operation 5607 the user device corrects the image and transmits the corrected image to an external server or another user device. When it is determined in operation 5606 that the image does not include the photographing restriction element, the user device transmits the image without correction in operation 5608.

57 is a method for correcting a captured image, according to an exemplary embodiment, to immediately correct the captured image without externally transmitting the captured image in the user device and to externally transmit the corrected captured image. A diagram of an example of an image.

Figure 57 illustrates a call between two smart phones 100-2a and 100-2b.

Here, the smart phones 100-2a and 1002-b do not store the video call screen unless requested. However, according to the current embodiment, since the photographing restriction element needs to be corrected even during the video call, the smart phone 100-2a can correct the image immediately without storing the image and transmit the corrected image to the smart phone 100. -2b. Referring to the video call screen 5710 displayed on the smart phone 100-2b, the face area 5712 of the person talking on the phone is not corrected, but the mosaic processing is performed on the face area 5711 of the photo restriction person.

58 is a method for correcting a captured image, in the case of not storing the captured image in a user device, and correcting the captured image and externally transmitting the corrected captured image, in accordance with an exemplary embodiment. A flow chart of an example of an image.

Referring to Figure 58, in operation 5801, the user device captures an image. Here, the user device can be any device capable of photographing, such as a smart phone, a video camera or a tablet PC.

In operation 5802, the user device identifies at least one of an individual included in the image, an object included in the image, and a location at which the image is captured. A method of identifying a person, object or location has been described above with reference to FIG.

In operation 5803, the user device determines whether the image includes a photographing restriction element based on at least one of the identified relationship between the individual and the user of the user device, the identified object, and the recognized location. A method of determining whether an image contains a photographing restriction element has been described above with reference to FIG.

When it is determined in operation 5804 that the user device includes a photography limiting element, the image is corrected in operation 5805 and then transmitted to an external server or another user device. When it is determined in operation 5804 that the user device does not include a photographing restriction element, the image is transmitted to the external server or another user device without modification in operation 5806.

59 to 61 are diagrams for describing an example of correcting an image captured at a photographing restriction position, according to an exemplary embodiment.

Referring to FIG. 59, user A's smart phone 100-2 can capture an image 5910a of an individual 5911 at a photography-capable location, such as a concert hall, museum, or military security zone, at a photography-capable location, such as a street or restaurant. The image 5920a of the individual 5911 is captured. The photography limit of the individual 5911 can be changed based on the photographing position. For example, if the individual 5911 is a celebrity, the photo located at the concert hall may be attributed to copyright restrictions, but the photos located on the street may be unrestricted. Therefore, the smart phone 100-2 transmits the image to the server 300 for image processing. Here, the smart phone 100-2 transmits the image along with the location information to the server 300. The location information is information about the location of the captured image. The smart phone 100-2 can determine the location by using the GPS included therein, and then transmit the location information to the server 300. Alternatively, the smart phone 100-2 generates location information via any of a variety of methods (e.g., by using a location of a base station or access point (AP) connected to the smart phone 100-2 to determine location).

Referring to FIG. 60, the server 300 can determine the position of the captured image by analyzing the position information received from the smart phone 100-2, and determine whether the position is the shooting restriction position. In other words, the server 300 determines whether the location is included in the pre-stored photography limit location database. Here, the pre-stored photography restriction location database may store photography restriction locations (such as concert halls, museums, or military security zones) depending on the user's photography restricted location or restricted by ordinary people.

When the position of the captured image 5910a is determined not to be the photographing limit position, the server 300 can transmit the uncorrected image 5910b to the smart phone 100-2. Alternatively, the server 300 can transmit the image 5910a to the smart phone 100-2 that does not require the corrected message to be transmitted.

When the position of the captured image 5920a is determined to be the photographing restriction position, the server 300 determines that the image 5920a includes the photographing restriction element. Therefore, the server 300 can correct a portion of the image 5920a corresponding to the photographing restriction individual. In detail, the server 300 can be generated by performing a mosaic process on a face region of the person 5911 in the image-restricting personal image 5920a, performing an alternative image-finishing process on the face region, or performing blurring processing on the face region. The corrected image 5920b is output.

In FIG. 59 and FIG. 60, the server 300 corrects the image. Alternatively, the server 300 can determine whether the position of the captured image is the photographing limit position and determine the position information by analyzing the position information received from the smart phone 100-2. The result is transmitted to the smart phone 100-2, and the smart phone 100-2 can correct the image.

Meanwhile, in FIGS. 59 and 60, the photographing restriction position differs depending on the subject, but alternatively, the photographing restriction position can be equally applied to all users. In other words, the photographing restriction position can be determined regardless of whether or not the individual is included in the image, and when the position of the captured image corresponding to the photographing restriction position is determined, a part of the image corresponding to the person or object included in the image can be corrected. .

Alternatively, as shown in FIG. 61, the smart phone 100-2 of the user A can determine the position of the captured image by analyzing the position information, and determine whether the position corresponds to the photographing limit position. In other words, the smart phone 100-2 determines whether the location is included in the photography restricted location database stored in the storage of the smart phone 100-2.

When it is determined that the position of the captured image 5910a is not the photographing restriction position of the individual 5911, the smart phone 100-2 can store the uncorrected image 5910b.

When it is determined that the position of the captured image 5920a is the photographing restriction position of the individual 5911, the smart phone 100-2 determines that the image 5920a includes the photographing restriction element. Therefore, the smart phone 100-2 can correct a part of the image 5920a corresponding to the individual 5911 for the individual who is limited to photography. In detail, the smart phone 100-2 can be generated and outputted by performing a mosaic process on the face area of the person 5911 in the image 5920a, performing an alternate image arrangement process on the face area, or performing blurring processing on the face area. Corrected image 5920b.

Similarly, when photographing an individual or an object, the individual or object may be limited or unrestricted based on the location of the photographic person or object.

FIG. 62 is a flowchart illustrating a method of correcting an image captured at a photographing restriction position, according to an exemplary embodiment.

Referring to Figure 62, in operation 6201, the user device captures an image. Here, the user device may be any of a general photographing device (such as a camera or a video camera), a wearable device (such as a smart eyeglass or a smart watch), and a mobile device (such as a smart phone or a tablet PC). Alternatively, or in the alternative, the user device can be any of a variety of devices capable of photography.

In operation 6202, the user device identifies the location of the user device during photography. Here, the user device can determine the location by using the GPS included therein. Alternatively, the user device can determine the location by using the location of the base station or AP connected to the user device.

In operation 6203, the server determines whether the position corresponds to the photographing limit position. Alternatively, the user device can determine if the location corresponds to a photography limited position. In detail, it is determined whether the position corresponds to a photographing restriction position of an individual included in the image, or a photographing restriction position of the subject. Thus, the server or user device can analyze the location information to determine the location of the captured image and compare the location to a photo-restricted location pre-stored in the photo-restricted location database. Here, the photography restriction location database can store photography restriction positions (such as concert halls, museums, and military security areas) according to people's photography restriction positions or restricted by ordinary people.

When the position corresponds to the photographing limit position, the server or user device may determine whether the image contains the photographing restriction element in operation 6204. In other words, when the location corresponds to a particular person's photography limited position, the server or user device can determine whether the image contains a particular individual; and when the location corresponds to a photography limited location of all people or objects, the server or user device can Determine if the image contains at least one person or object. When the image includes a photography limiting component, the server can correct the image and transmit the corrected image to the user device in operation 6205. Alternatively, the user device can correct the image and store the corrected image. For example, the server or the user device can generate the corrected image by performing a mosaic process, a blur process, or an alternate image layout process on the image.

63-66 are diagrams for describing an example of correcting a captured image according to a user selection, in accordance with an exemplary embodiment.

Referring to FIG. 63, the smart phone 100-2 of the user A captures an image including the user B and the user C. Subsequently, the smart phone 100-2 displays the image on the screen.

In FIGS. 64 and 65, when the user A selects an individual to be subjected to the protection process from the image via a different method, a part of the image corresponding to the selected individual is corrected. Here, the user A may be provided on the smart phone 100-2, for example, as a preview screen or a temporary screen before being stored in the storage of the smart phone 100-2 or shared with the external device. Choose an individual.

Referring to FIG. 64A, the smart phone 100-2 recognizes the faces of the user B and the user C by performing face recognition on the original image 6400a. Subsequently, the smart phone 100-2 can provide a result indicating facial recognition of the face of the user B and the user C. The result can be provided as visible feedback, for example, by highlighting (flashing or coloring) the contours of the faces or faces of User B and User C. Subsequently, when the user A selects one of the user B and the user C, for example, when the user C is selected, the smart phone 100-2 can perform the protection process on the user C. For example, the smart phone 100-2 can correct a portion of the image corresponding to the user C such that the user C is indistinguishable. In detail, as shown in FIG. 64B, the smart phone 100-2 can display the corrected result by performing mosaic processing, substitute image arrangement processing, or blurring processing on the face area of the user C in the original image 6400a. Image 6400b.

As another example, referring to FIG. 65A, the smart phone 100-2 can recognize the faces of the user B and the user C by performing face recognition on the original image. Subsequently, the smart phone 100-2 can determine whether the face corresponds to the photography restriction individual. When it is determined that the user B and the user C correspond to the photography restriction person, the smart phone 100-2 can perform the protection process on the user B and the user C. For example, the smart phone 100-2 can correct the portion of the original image corresponding to the user B and the user C, so that the user B and the user C are indistinguishable. In detail, the smart phone 100-2 can display the corrected image 6500a obtained by performing mosaic processing, substitute image editing processing, or blurring processing on the face regions of the user B and the user C who restrict people for photography. Then, when the user A selects one of the user B and the user C, for example, when the user B is selected, the smart phone 100-2 can determine that the user B is not to be protected. Therefore, the smart phone 100-2 can correct a portion of the corrected image 6500a corresponding to the user B so that the user B is discernible. In detail, as shown in FIG. 65B, the smart phone 100-2 can be displayed by releasing the mosaic processing, the alternative image arrangement processing, or the blurring processing on the face area of the user B executed in the corrected image 6500a. Corrected image 6500b.

As another example, referring to FIG. 66A, the smart phone 100-2 recognizes the faces of the user B and the user C by performing face recognition on the original image 6600a. Subsequently, the smart phone 100-2 can provide a pop-up screen 6610 that asks the user A whether to perform protection processing on the user C. When the user A selects a button for requesting the smart phone 100-2 to perform the protection process via the popup screen 6610, the smart phone 100-2 can perform the protection process on the user C. Subsequently, as shown in FIG. 66B, the smart phone 100-2 can display the corrected image 6600b obtained by performing mosaic processing, substitute image arrangement processing, or blurring processing on the face area of the user C in the original image 6600a. .

Before the image captured by the smart phone 100-2 is stored in the storage of the smart phone 100-2, the screen for selecting the person on whom the protection process is to be performed can be displayed. Alternatively, the screen may be displayed before the smart phone 100-2 provides the image to an external device such as an external server or a third party smart phone.

Meanwhile, when the individual on whom the protection process is to be performed is selected, information about the individual can be stored in the database or server 300 of the smart phone 100-2. For example, an individual may be designated as a photography restriction element in the information. In this case, when the individual is photographed in a future image, the smart phone 100-2 can automatically determine that the individual is a photography restriction individual. Subsequently, the smart phone 100-2 can perform mosaic processing, substitute image arrangement processing, or blur processing on an area corresponding to a future image of the individual.

Also, when the smart phone 100-2 or the server 300 does not have information on the face included in the captured image, it is useful to restrict the individual by the user selecting the photography. In other words, when the smart phone 100-2 or the server 300 does not have information about the face included in the captured image, the smart phone 100-2 or the server 300 can process the captured image inconsistently (eg, protect The face may not be protected, but the face may be determined according to the user's choice, thus increasing the degree of freedom of the user for processing the captured image.

67 and 68 are flowcharts illustrating a method of correcting a captured image based on user selection, in accordance with an illustrative embodiment.

Referring to Figure 67, in operation 6701, the user device captures an image containing the individual. Here, the user device can be any device capable of photographing, such as a smart phone, a video camera or a tablet PC.

In operation 6702, the user device performs facial recognition on an individual included in the image.

In operation 6703, the user device can provide a result of facial recognition. For example, the user device can display a screen in which visual feedback is applied to the individual.

In operation 6704, when the screen is displayed, the user of the user device can select the individual on whom the protection process is to be performed.

When the user selects an individual on whom the protection process is to be performed, the user device may perform a protection process on the selected individual in operation 6705. For example, the user device can correct the image such that the selected individual is indistinguishable.

As another example, referring to FIG. 68, the user device captures an image in operation 6801.

In operation 6802, the user device performs facial recognition on an individual included in the image.

In operation 6803, the user device can determine whether the individual is a photography restriction individual. When the individual restricts the individual for photography, the user device can perform the protection process on the individual in operation 6804. When the individual is not a photography-restricted individual, the user device may not perform the protection process.

In operation 6805, the user device can display the person on whom the protection process is performed. For example, the user device may perform a mosaic process, an alternate image layout process, or a blur process on an area corresponding to the image of the individual.

In operation 6806, when the personal display on which the protection process is performed is displayed, the user can select the self-protection to process the individual to be released.

When the self-protection process is to be released, the user device may remove the protection process from the selected individual in operation 6807. For example, the user device may revoke mosaic processing, alternate image layout processing, or blurring processing performed on an area corresponding to the image of the selected individual.

As described above, in accordance with one or more exemplary embodiments, when an image containing a photography-restricted individual, object, or location is captured by a user device, all of the image may be corrected such that photography limits the individual, object, or location. It is indistinguishable, so protecting photography limits the privacy of individuals and prevents individuals from obtaining images that contain photography-restricted objects or locations for security or copyright reasons.

When not limited thereto, the illustrative embodiments can be embodied as computer readable code on a computer readable recording medium. A computer readable recording medium is any data storage device that can store data that can be subsequently read by a computer system. Examples of computer readable recording media include read-only memory (ROM), random-access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices. The computer readable recording medium can also be distributed over a computer system coupled to the network to store and execute the computer readable code in a distributed manner. Furthermore, the illustrative embodiments can be written as a computer program transmitted in a computer readable transmission medium (such as a carrier wave) and can be received and implemented in a general purpose or special purpose computer for executing the program. In addition, it should be appreciated that in an exemplary embodiment, one or more of the above described devices and apparatus can include circuitry, a processor, a microprocessor, etc., and can execute a computer stored in a computer readable medium. Program.

The above exemplary embodiments are merely illustrative and are not to be considered as limiting. The teachings of the present invention can be readily applied to other types of devices. In addition, the description of the exemplary embodiments is intended to be illustrative, and is not intended to

1‧‧‧Users
2‧‧‧Users
3‧‧‧Users
3-1‧‧‧ edge
20‧‧‧ statue
20-1‧‧‧ edge
30‧‧‧No photography signs
30-1‧‧‧ symbol
30-2‧‧‧Words
100‧‧‧Smart glasses
100-1‧‧‧Smart Watch
100-2‧‧‧Smart Phone
100-2a‧‧‧Smart Phone
100-2b‧‧‧Smart Phone
100-3‧‧‧ camera
100-4‧‧‧Video Recorder
100-5‧‧‧ Tablet PC
200‧‧‧Smart Phone
201‧‧‧ operation
202‧‧‧ operation
203‧‧‧ operation
204‧‧‧ operation
205‧‧‧ operation
300‧‧‧Server
400a‧‧‧ original image
400b‧‧‧Fixed imagery
500a‧‧‧ original image
500b‧‧‧Fixed imagery
600a‧‧‧ original image
600b‧‧‧Fixed imagery
700a‧‧‧ original image
700b‧‧‧Fixed imagery
800a‧‧‧ original image
800b‧‧‧Fixed imagery
801‧‧‧ pixels
802‧‧ pixels
900a‧‧‧ original image
900b‧‧‧Fixed imagery
1000‧‧‧User device
1001‧‧‧Alternative imagery
1000a‧‧‧ original image
1000b‧‧‧Fixed imagery
1100‧‧‧User interface
1100a‧‧‧ original image
1100b‧‧‧Fixed imagery
1110‧‧‧Alternative imagery
1200‧‧‧ camera
1200a‧‧‧ original image
1200b‧‧‧Fixed imagery
1210‧‧‧Alternative imagery
1300‧‧‧Storage
1301‧‧‧ operation
1302‧‧‧ operation
1303‧‧‧ operation
1304‧‧‧ operation
1305‧‧‧ operation
1400‧‧‧Communication interface
1401‧‧‧ operation
1402‧‧‧ operation
1403‧‧‧ operation
1404‧‧‧ operation
1500‧‧‧ controller
1501‧‧‧ operation
1502‧‧‧ operation
1503‧‧‧ operation
1504‧‧‧ operation
1510‧‧‧Image Analyzer
1520‧‧‧Image Corrector
1600a‧‧‧ original image
1600b‧‧‧Fixed imagery
1601‧‧‧ operation
1602‧‧‧ operation
1603‧‧‧ operation
1604‧‧‧ operation
1605‧‧‧ operation
1800a‧‧‧ original image
1800b‧‧‧Fixed imagery
1901‧‧‧ operation
1902‧‧‧ operation
1903‧‧‧ operation
1904‧‧‧ operation
1905‧‧‧ operation
2301‧‧‧ operation
2302‧‧‧ operation
2303‧‧‧ operation
2304‧‧‧ operation
2305‧‧‧ operation
2306‧‧‧Operation
2307‧‧‧ operation
2401‧‧‧ operation
2402‧‧‧ operation
2403‧‧‧ operation
2404‧‧‧ operation
2405‧‧‧ operation
2406‧‧‧ operation
2600a‧‧‧ original image
2600b‧‧‧Fixed imagery
2701‧‧‧ operation
2702‧‧‧ operation
2703‧‧‧ operation
2704‧‧‧Operation
2705‧‧‧ operation
2706‧‧‧ operation
2707‧‧‧ operation
3201‧‧‧ operation
3202‧‧‧Operation
3203‧‧‧ operation
3204‧‧‧ operation
3205‧‧‧ operation
3206‧‧‧ operation
3207‧‧‧ operation
3400a‧‧‧ original image
3400b‧‧‧Fixed imagery
3500a‧‧‧ original image
3500b‧‧‧Fixed imagery
3600a‧‧‧ original image
3600b‧‧‧Fixed imagery
3601‧‧ ‧ pixels
3602‧‧ ‧ pixels
3700a‧‧‧ original image
3700b‧‧‧Fixed imagery
3710‧‧‧Alternative imagery
3801‧‧‧ operation
3802‧‧‧ operation
3803‧‧‧ operation
3804‧‧‧ operation
3805‧‧‧ operation
3806‧‧‧Operation
4000a‧‧‧ original image
4000b‧‧‧Fixed imagery
4100a‧‧‧ original image
4100b‧‧‧Fixed imagery
4201‧‧‧ operation
4202‧‧‧ operation
4203‧‧‧ operation
4204‧‧‧ operation
4400a‧‧‧ original image
4400b‧‧‧Fixed imagery
4500a‧‧‧ original image
4500b‧‧‧Fixed imagery
4601‧‧‧ operation
4602‧‧‧ operation
4603‧‧‧ operation
4604‧‧‧ operation
4710‧‧‧ Privacy Protection Information
5001‧‧‧ operation
5002‧‧‧ operation
5003‧‧‧ operation
5004‧‧‧ operation
5005‧‧‧ operation
5006‧‧‧ operation
5007‧‧‧ operation
5210‧‧‧Image
5211‧‧‧Area
5220‧‧‧Enlarge image
5221‧‧‧Area
5301‧‧‧ operation
5302‧‧‧ operation
5303‧‧‧ operation
5304‧‧‧ operation
5305‧‧‧ operation
5306‧‧‧Operation
5410‧‧‧Image
5411‧‧‧Area
5420‧‧‧Enlarge image
5421‧‧‧Area
5510‧‧‧Image
5520‧‧ images
5521‧‧‧Area
5530‧‧‧"Send" button
5601‧‧‧ operation
5602‧‧‧ operation
5603‧‧‧Operation
5604‧‧‧ operation
5605‧‧‧ operation
5606‧‧‧Operation
5607‧‧‧ operation
5608‧‧‧ operation
5710‧‧‧Video Call Screen
5711‧‧‧Face area
5712‧‧‧Face area
5801‧‧‧ operation
5802‧‧‧ operation
5803‧‧‧ operation
5804‧‧‧ operation
5805‧‧‧ operation
5806‧‧‧ operation
5910a‧‧‧Image
5910b‧‧‧Image
5911‧‧‧person
5920a‧‧ images
5920b‧‧‧Image
6201‧‧‧ operation
6202‧‧‧ operation
6203‧‧‧ operation
6204‧‧‧Operation
6205‧‧‧ operation
6400a‧‧‧ original image
6400b‧‧‧Fixed imagery
6500a‧‧‧ original image
6500b‧‧‧Fixed imagery
6600a‧‧‧ original image
6600b‧‧‧Fixed imagery
6610‧‧‧ pop-up screen
6701‧‧‧ operation
6702‧‧‧Operation
6703‧‧‧ operation
6704‧‧‧Operation
6705‧‧‧ operation
6801‧‧‧ operation
6802‧‧‧ operation
6803‧‧‧Operation
6804‧‧‧ operation
6805‧‧‧ operation
6806‧‧‧Operation
6807‧‧‧Operation
A‧‧‧ user
B‧‧‧Users
C‧‧‧Users

The above and/or other aspects will be more apparent from the description of the exemplary embodiments, which are illustrated in the accompanying drawings in which: FIG. Figure. 2 is a flow chart of a method of modifying an image containing a photographic limiting element in accordance with an illustrative embodiment. 3 through 6 are diagrams for describing a method of correcting an image including a photographing restriction element, according to an exemplary embodiment. 7 through 12 are diagrams for describing a method of correcting a personal image including a photography restriction, according to an exemplary embodiment. 13-16 are flow diagrams of a method of modifying a photo-restricted personal image in accordance with an illustrative embodiment. 17 and 18 are diagrams for describing a method of correcting an image including a photographing restriction element by using at least two user devices, according to an exemplary embodiment. 19 is a flow diagram of a method of modifying an image containing a photographic restriction element using at least two user devices, in accordance with an illustrative embodiment. 20 to 22 are diagrams for describing a method of correcting an image including a photographing restriction element via communication with a device of a subject, according to an exemplary embodiment. 23 and 24 are flowcharts of a method of correcting an image including a photographing restriction element via communication with a device of a subject, in accordance with an exemplary embodiment. 25 and 26 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server, according to an exemplary embodiment. 27 is a flow chart of a method of correcting an image containing a photographing restriction element by using a server, in accordance with an exemplary embodiment. 28 through 31 are diagrams for describing a method of correcting an image including a photographing restriction element by using a server, according to other exemplary embodiments. 32 is a flow chart of a method of correcting an image containing a photographing restriction element by using a server, according to another exemplary embodiment. 33 to 37 are diagrams for describing a method of correcting an image including a photographing restriction object, according to an exemplary embodiment. 38 is a flow chart of a method of modifying an image containing a photographic restriction object, in accordance with an illustrative embodiment. 39 to 41 are diagrams for describing a method of correcting an image captured at a photographing restriction position, according to an exemplary embodiment. 42 is a flow chart of a method of correcting an image captured at a photographing limit position, in accordance with an exemplary embodiment. 43 to 45 are diagrams for describing a method of correcting an image captured at a photographing restriction position, according to other exemplary embodiments. FIG. 46 is a flowchart of a method of correcting an image captured at a photographing restriction position, according to another exemplary embodiment. 47 to 49 are diagrams for describing an example of restricting personal photography according to privacy protection information, according to an exemplary embodiment. FIG. 50 is a flowchart of a method of restricting personal photography based on privacy protection information, according to an exemplary embodiment. Figure 51 is a block diagram of a user device in accordance with an illustrative embodiment. FIG. 52 is a diagram for describing an example of correcting a captured image before storing the captured image in a user device, in accordance with an exemplary embodiment. 53 is a flowchart depicting an example of correcting a captured image prior to storing the captured image in a user device, in accordance with an exemplary embodiment. 54 and 55 are diagrams for explaining an example of correcting a captured image, and correcting an image stored in the user device while externally transmitting the image, according to an exemplary embodiment. FIG. 56 is a flowchart for explaining an example of correcting a captured image, and correcting an image stored in the user device while externally transmitting the image, according to an exemplary embodiment. 57 is a method for correcting a captured image, according to an exemplary embodiment, to immediately correct the captured image without externally transmitting the captured image in the user device and to externally transmit the corrected captured image. A diagram of an example of an image. 58 is a method for correcting a captured image, in the case of not storing the captured image in a user device, and correcting the captured image and externally transmitting the corrected captured image, in accordance with an exemplary embodiment. A flow chart of an example of an image. 59 to 61 are diagrams for describing an example of correcting an image captured at a photographing restriction position, according to an exemplary embodiment. FIG. 62 is a flowchart illustrating a method of correcting an image captured at a photographing restriction position, according to an exemplary embodiment. 63, 64A, 64B, 65A, 65B, 66A, and 66B are diagrams for describing an example of correcting a captured image according to a user selection, according to an exemplary embodiment. 67 and 68 are flowcharts illustrating a method of correcting a captured image based on user selection, in accordance with an illustrative embodiment.

1‧‧‧Users

2‧‧‧Users

20‧‧‧ statue

30‧‧‧No photography signs

100‧‧‧Smart glasses

100-1‧‧‧Smart Watch

100-2‧‧‧Smart Phone

100-3‧‧‧ camera

100-4‧‧‧Video Recorder

100-5‧‧‧ Tablet PC

A‧‧‧ user

B‧‧‧Users

Claims (15)

A user device comprising: a memory configured to store data; and a controller configured to identify an image included in the user device or captured by the user device from an external source An individual, and based on the information stored in the storage, in response to the degree of intimacy between the identified individual and the user of the user device being below a predetermined level, A portion of the image of the identified individual is changed to be unrecognizable. The user device of claim 1, further comprising a camera configured to capture the image. The user device of claim 1, wherein the controller is further configured to determine whether a position at which the image is captured is a photographic limited position, and when the position is the photographic limited position The portion of the image corresponding to the identified individual changes to be unrecognizable. The user device of claim 1, wherein the material stored in the storage is an address book, and the controller is further configured to respond to the identified individual not including The degree of intimacy is determined to be lower than the predetermined level in the address book. The user device of claim 1, wherein the data stored in the storage is at least one photo, and the controller is responsive to the identified individual not included in the at least The degree of the intimacy between the user and the identified individual is determined to be lower than the predetermined level in a photo. The user device of claim 1, wherein the data stored in the storage is a social network service usage record of the user, and the controller A social network service usage record is used to determine the degree of intimacy. The user device of claim 1, wherein the controller determines whether the location of capturing the image is a photography restriction position by accessing a database of the external server, and the location is The image is corrected when the shooting limit position is described. The user device of claim 1, wherein the controller is further configured to identify a photography restriction object from the image and determine a capture in response to recognizing the photography restriction object from the image The position of the image is the photography limit position. The user device of claim 1, wherein the controller performs at least one of a mosaic process, a blur process, a background image arrangement process, and an alternate image arrangement process on the portion of the image, To change the portion of the image to be unrecognizable. A system for correcting images, the system comprising: a server configured to perform image processing; and a user device configured to request the image processing when transmitting the image to a server; The server is further configured to identify an individual included in the image received from the user device and to respond to a low degree of intimacy between the identified individual and a user of the user device A portion of the image corresponding to the identified individual is changed to unrecognizable at a predetermined level and the image is transmitted to the user device. The system for correcting an image according to claim 10, wherein the server determines whether the position of capturing the image is a photographing restriction position by using the stored data, and responding to the position as a location The photographing limits the position and changes the portion of the image corresponding to the recognized individual to be unrecognizable. The system for correcting an image according to claim 10, wherein the server receives a time and a position of capturing the image from the user device, and determines the location by using the stored data. Whether the recognized individual's photography is limited at the received time and the received location. A system for correcting images as described in claim 10, wherein the user device encodes the image and transmits the encoded image by using an encryption key received from the server. To the server, and the server decodes the encoded image and changes the portion of the decoded image corresponding to the identified individual. A user device comprising: a camera configured to capture an image; a communication interface configured to transmit a photographic notification to a device captured by the subject in the image, and from the subject's location The device receives a response to the photography notification; and a controller configured to modify the image based on the response indicating whether the capturing of the image is permitted. A method for correcting an image by a user device, the method comprising: identifying an individual included in the image; determining whether a degree of intimacy between the identified individual and a user of the user device is low At a predetermined level; and changing a portion of the image corresponding to the identified person to be unrecognizable in response to the degree of intimacy being lower than the predetermined level.