KR102596157B1 - Photo management device and method thereof - Google Patents

Abstract

This specification analyzes the images included in the stored photos to determine whether they are unnecessary photos, recommends deletion of the photos if they are unnecessary photos, lists photos recommended for deletion, and provides a display for selecting whether to delete the photos. A photo management device and method for printing photos are disclosed.

Description

Photo management device and method {PHOTO MANAGEMENT DEVICE AND METHOD THEREOF}

This specification provides a photo management device and method for managing photos.

Today, taking photos using a smartphone has become an essential part of modern life. In response to these social changes, modern people require larger storage capacity and the number of unnecessary photos is increasing. To support storage, cloud-based photo storage services have emerged from various platforms and many people are using these services. As the number of photos increases, it becomes difficult for users to find the photos they want and it takes a lot of time to organize them.

With each new smartphone released, features are improving to help people take better photos, making it possible for anyone to take photos as good as a professional photographer. Meanwhile, the resolution of photos has become increasingly larger and the size of one photo has grown to 10MB. In everyday life, people take a lot of photos, and especially when they travel, they take even more photos. Therefore, photo management is becoming more necessary and important.

Platform companies such as Naver, Google, and Microsoft provide cloud services for photo management. People are using these services to back up. However, as the number of photos increases, various problems arise.

First, storage capacity becomes insufficient. Second, it becomes difficult to find the photo you want. Third, there are many unnecessary photos, such as shaky photos. In particular, the third problem is causing insufficient storage capacity and difficulty in photo retrieval.

Existing photo management services primarily provide storage with some search and sorting capabilities, but do not provide a solution to the third problem.

These embodiments can provide a photo management device and method that classifies photos using deep learning and recommends unnecessary photos to facilitate photo management.

Additionally, the present embodiments can provide a photo management device and method that provide a function of identifying a subject and tagging a photo, a function of checking incorrectly taken photos, and a function of recommending similar photos.

Additionally, the present embodiments can provide a photo management device and method that allow users to easily manage photos and efficiently use storage capacity.

In one aspect, a photo management device according to an embodiment includes an input unit, a communication unit that exchanges data with another device through a communication network, a memory unit that stores photos, and analyzes images included in stored photos to determine whether they are unnecessary photos. It may include a control unit that analyzes the photo and recommends deletion of the photo if it is an unnecessary photo, and an output unit that lists the photos recommended for deletion by the control unit and outputs a display for selecting whether to delete the photos.

In another aspect, a photo management method according to another embodiment includes the steps of storing photos, analyzing images included in the stored photos to determine whether they are unnecessary photos, and recommending deletion of the photos if they are unnecessary photos, and deleting the photos. It may include listing the recommended photos and outputting a display for selecting whether to delete the photos.

The photo management device and method according to the present embodiments can classify photos using deep learning and recommend unnecessary photos to facilitate photo management.

Additionally, the photo management device and method according to the present embodiments can provide a function of identifying the subject and tagging the photo, a function of checking an incorrectly taken photo, and a function of recommending similar photos.

Additionally, the photo management device and method according to the present embodiments allow users to easily manage photos and use storage capacity efficiently.

1 is a block diagram of a photo management device according to an embodiment.
Figure 2 is a data structure diagram of the memory unit.
Figure 3 is a block diagram of the control unit.
Figure 4 is a configuration diagram of a deep learning model used in the photo object recognition module of Figure 3.
Figure 5 shows the process by which the blurred photo detection module determines the blurriness of the photo using Laplacian distribution.
Figure 6 is a configuration diagram of a deep learning model used in the dark photo detection module of Figure 3.
Figures 7a and 7b show the connection states of all nodes when the deep learning model used in the dark photo detection module of Figure 3 applies the dropout technology and when it does not.
Figure 8 is a flowchart of a photo management method according to another embodiment.
Figure 9 is a flowchart of steps for recommending deletion of the unnecessary picture in Figure 8.
Figure 10 is a block diagram of a cloud-based photo management system.
Figure 11 shows a model learning process according to an embodiment.
Figure 12 is a partial image of a smartphone application provided with photo management services from a cloud-based photo management system.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to illustrative drawings. In adding reference numerals to components in each drawing, the same components may have the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present technical idea, the detailed description may be omitted. When “comprises,” “has,” “consists of,” etc. mentioned in the specification are used, other parts may be added unless “only” is used. When a component is expressed in the singular, it can also include the plural, unless specifically stated otherwise.

Additionally, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, order, or number of the components are not limited by the term.

In the description of the positional relationship of components, when two or more components are described as being “connected,” “coupled,” or “connected,” the two or more components are directly “connected,” “coupled,” or “connected.” ", but it should be understood that two or more components and other components may be further "interposed" and "connected," "combined," or "connected." Here, other components may be included in one or more of two or more components that are “connected,” “coupled,” or “connected” to each other.

In the description of temporal flow relationships related to components, operation methods, production methods, etc., for example, temporal precedence relationships such as “after”, “after”, “after”, “before”, etc. Or, when a sequential relationship is described, non-continuous cases may be included unless “immediately” or “directly” is used.

On the other hand, when a numerical value or corresponding information (e.g., level, etc.) for a component is mentioned, even if there is no separate explicit description, the numerical value or corresponding information is related to various factors (e.g., process factors, internal or external shocks, It can be interpreted as including the error range that may occur due to noise, etc.).

Throughout the detailed description and claims of the present invention, 'learning' or 'learning' is a term that refers to performing machine learning through procedural computing, and refers to mental actions such as human educational activities. Those skilled in the art will understand that it is not intended to do so.

Embodiments of the present invention will be described in detail below with reference to the drawings.

1 is a block diagram of a photo management device according to an embodiment.

Referring to FIG. 1, the photo management device 10 according to one embodiment manages a large amount of photos by finding unnecessary photos among many stored photos and recommending deletion, thereby enabling efficient use of storage space. support

The photo management device 10 according to one embodiment provides a photo management service that facilitates photo management by classifying photos using deep learning and recommending unnecessary photos. This service provides functions for tagging photos by identifying the subject, checking for incorrectly taken photos, and recommending similar photos.

This service allows users to easily manage photos and use storage capacity efficiently.

Specifically, the photo management device 10 analyzes the images included in the stored photos to determine whether they are unnecessary photos, recommends deletion of the photos if they are unnecessary, lists photos recommended for deletion, and deletes the photos. Outputs a display indicating whether or not to select.

The photo management device 10 may be a communication terminal or a server connected to the communication terminal and a communication network, for example, a cloud server. The photo management device 10 may refer to a communication system in which some of the components described later are included in a communication terminal and others are included in a server.

Communication terminals described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, and slate PCs. , tablet PCs, ultrabooks, wearable devices, etc. may be included. Wearable devices may be, for example, a watch-type terminal (smartwatch), a glass-type terminal (smart glass), a head mounted display (HMD), etc.

The photo management device 10 includes an input unit 12 for inputting information, a communication unit 14 for exchanging data with other devices through a communication network, a memory unit 16 for storing photos, and an analysis and analysis of images included in the stored photos. A control unit 18 that analyzes whether a photo is unnecessary and recommends deletion of the photo if it is an unnecessary photo, and an output unit that lists photos recommended for deletion by the control unit 18 and outputs a display for selecting whether to delete the photos. Includes (20).

The input unit 12 may be, for example, a photographing device that captures images, such as a camera. The communication network may be a wired communication network or a wireless communication network. Accordingly, the communication unit 14 may be capable of wired communication, wireless communication, or wired and wireless communication.

For example, a photo stored in the memory unit 16 may be a photo input through the input unit 12. As another example, the photo stored in the memory unit 16 may be a photo received by the communication unit 14 from another device through a communication network. For example, if the photo management device 10 is a cloud server, photos uploaded from the users' communication terminals through a communication network can be stored for each user in the memory unit 16.

Unnecessary photos are photos that the user determines are not necessary to keep stored. For example, unnecessary photos include incorrectly taken photos such as shaky/dark photos, similar photos taken multiple times of the same subject, and photos captured on screen for information storage purposes (e.g. screenshots). etc. are included.

Figure 2 is a data structure diagram of the memory unit.

Referring to Figures 1 and 2, the memory unit 16 manages photos and provides a space to store the user's photos. The memory unit 16 may store meta information of the photo (eg, latitude/longitude, date, time, file name, etc.) and the location of the photo. If the photo management device 10 is a cloud server, the memory unit 16 may be configured as a database. When the photo management device 10 is a communication terminal, the memory unit 16 may be a general memory device managed by a file management program included in the operating system.

For example, as shown in FIG. 2, the memory unit 16 includes a user table, a photo meta information table, a photo tag table, and a deletion information table. information table), but is not limited thereto.

The user table can store user ID and email.

The photo meta information table can store meta information provided by the communication terminal, a boolean value indicating whether deletion is possible, and whether a screenshot is taken. The photo meta information table is used in the similar photo search process and screenshot search process. The value of the deleteable field is used in the deletion recommendation function.

A Boolean value indicating what the subject is is stored in the photo tag table, and is automatically recorded when the user saves the photo to the memory. Tag information is used in the process of searching photos and classifying similar photos.

A Boolean value indicating the reason for photo deletion is stored in the deletion information table. This table stores the photo analysis result values analyzed by the control unit 18.

Figure 3 is a block diagram of the control unit.

Referring to FIG. 3, the control unit 18 analyzes the images included in the photos stored in the memory unit 16 described with reference to FIG. 2 to determine whether the photos are unnecessary, and if the photos are unnecessary, recommends deletion of the photos. and the result is stored in the memory unit 16.

The control unit 18 includes a photo object recognition module 26 that identifies objects in the photo, a shaken photo detection module 28 that determines whether the photo is shaken, and a dark photo detection module 30 that determines whether the photo is dark. ), a similar photo detection module 32 that searches for similar photos among photos, and a screen shot search module 33 that searches for screenshots among photos.

The control unit 18 includes all of a photo object recognition module 26, a shaken photo detection module 28, a dark photo detection module 30, a similar photo detection module 32, and a screenshot search module 33, or these Only some of them can be included selectively or fixedly.

In addition, the control unit 18 includes a photo object recognition module 26, a shaken photo detection module 28, a dark photo detection module 30, a similar photo detection module 32, and a screenshot search module 33. Depending on the user's choice, some of the modules may be disabled. For example, if a specific user selects only blurry photos and dark photos as unnecessary photos, the control unit 18 may disable the similar photo detection module 32.

The photographic object recognition module 26 divides the image included in the photograph into several grids, each grid predicts a certain number of bounding boxes to identify the location of the object, and a confidence score is assigned to each bounding box. , identify objects in the photo using each predicted bounding box and confidence score.

The object detection process of the photo object recognition module 26 is specifically as follows. The input image is divided into several grids. When the center of an object is in a specific grid, that grid plays a role in detecting the specific object. Each grid predicts a certain number of 'bounding boxes' to identify the location of an object, and each bounding box is assigned a 'confidence score'.

A neural network is used to predict the conditional class probability of the bounding box and the location of the corresponding object. The confidence score is the IOU (Intersection Over Union) value between the predicted bounding box and the actual object boundary. When this value is equal to or greater than the predicted value, the object is determined to be included. As in existing image recognition systems, you can classify objects and determine which object has the highest probability to find out what the object is.

The photo object recognition module 26 may be comprised of software or hardware that identifies objects in a photo using each predicted bounding box and confidence score, but may also be configured using a deep learning model, as will be described later.

In this specification, a deep learning model may be a multi-layered artificial neural network. In other words, a deep learning model automatically learns the features of each image by learning a large amount of data in a deep neural network consisting of a multi-layer network, and through this, the objective function, i.e. This is a form of training the network in a way that minimizes the error in prediction accuracy.

In this specification, the deep learning model may use, for example, CNN (Convolutional Neural Network), DHN (Deep Hierarchical Network), CDBN (Convolutional Deep Belief Network), DDN (Deconvolutional Deep Network), etc., but currently Alternatively, various deep learning models may be used in the future. In this specification, the use of a CNN-based deep learning model is exemplarily described, but the present invention is not limited thereto and may use various deep learning models currently or in the future.

The screen shot search module 33 can search for a screenshot among photos using whether or not the screenshot is included in a photo meta information table that is stored together when the photo is saved by the camera. In the case of a screenshot, the camera module checks whether the screenshot is taken and stores it in the screenshot field of the photo meta information table of the memory unit 16. Therefore, the screenshot search module 33 can check whether a screenshot is taken using the photo meta information table. This may be similar to using a photo meta information table in the similar photo detection module 32.

Figure 4 is a configuration diagram of a deep learning model used in the photo object recognition module of Figure 3.

Referring to FIG. 4, the photo object recognition module 26 utilizes a deep learning model 31 consisting of multiple convolution layers, pooling layers, and fully connected layers to recognize photos. Objects can be identified.

The convolution layer creates a feature map for each area of the image using multiple filters. The pooling layer or subsampling layer reduces the size of the feature map and extracts features for the image that are invariant to changes in position or rotation.

The multiple convolution layer and pooling layer 33 repeats convolution and/or pooling to create various levels of features in the image, from low-level features such as points, lines, and faces to complex and meaningful high-level features. It can be extracted.

The fully connected layer 34 outputs a result value considering the relationships of the entire image.

The deep learning model 31 that constitutes the photo object recognition module 26 is trained using learning data, is tested, and is finally used to identify objects in the photo.

If the photo management device 10 is a communication terminal, the photo object recognition module 26 including the deep learning model learned from the server can be downloaded, stored in the communication terminal, and executed. The deep learning model 31 stored in the communication terminal may be periodically or aperiodically updated with a deep learning model that has undergone additional learning on the server.

Referring again to FIG. 3, the shaken photo detection module 28 determines whether the photo is blurred by using a method using a Gaussian basis filter, a method of measuring the degree of blur at edges, an edge sharpness analysis method, and Laplacian variance. can be judged.

When the blurred photo detection module 28 determines the blurriness of the photo using Laplacian dispersion, the blurriness of the photo can be determined by converting the photo to black and white and then calculating the Laplacian dispersion.

Figure 5 shows the process by which the blurred photo detection module determines the blurriness of the photo using Laplacian distribution.

Referring to FIG. 5, when the blurred photo detection module 28 determines the blurriness of the photo using Laplacian distribution, the input image 35 included in the photo is converted to black and white to create a black and white image 36. By calculating the Laplacian variance, the Laplacian variance value (37) can be obtained. The blurred photo detection module 28 can determine the blurriness of the photo using the Laplian variance value 37.

Figure 6 is a configuration diagram of a deep learning model used in the dark photo detection module of Figure 3.

Referring to FIG. 6, the dark photo detection module 30 may utilize a CNN-based deep learning model 38 to determine whether the photo is dark. As described above, in this specification, the use of a CNN-based deep learning model is exemplarily described, but the present invention is not limited thereto and may use various deep learning models currently or in the future.

For example, as shown in Figure 6, the dark photo detection module 30 is a deep learning model consisting of multiple convolution layers, a pooling layer 40, and a fully connected layer 42. (38) can be used to determine whether the photo is dark.

The deep learning model 38 aims to optimally learn the parameters present in each individual layer. The deep learning model 38 can apply random sampling (random data order) and regularization techniques. Random sampling means that the order of learning data learned in the training data set is different.

Normalization technology is a technology that reduces overfitting, which reduces the accuracy of deep learning models that are overly trained on training data containing noise. The normalization technology may be, for example, a drop-out technology or a drop-connected technology.

Dropout technology is a method of probabilistically excluding (excluding) specific nodes and learning them. Drop connected technology is a method of dropping and learning connections between nodes. Below, the dropout technique is described as an example of a regularization technique, but it may be any technique or algorithm that reduces overfitting currently or in the future.

The deep learning model (38) used in the dark photo detection module (30) is trained by removing nodes in each layer with a certain probability when learning with training data, and when learning is completed and testing, everything is restored to its original state. A dropout technique can be applied by considering the nodes and multiplying the weight each node has by the above probability.

Figures 7a and 7b show the connection states of all nodes when the deep learning model used in the dark photo detection module of Figure 3 applies the dropout technology and when it does not. In Figure 7b refers to the nodes removed from each layer.

Referring to FIGS. 7A and 7B, when learning with training data, the deep learning model 38 used in the dark photo detection module 30 trains the nodes in each layer by removing them with a specific probability p, and learns When this is completed and the actual image is classified, the dropout technique can be applied by considering all nodes as usual and multiplying the probability p by the weight each node has.

Dropout technology can prevent overfitting, in which learning is biased toward the learning data, by randomly removing nodes in each layer when the deep learning model 38 is learning, thereby disrupting learning.

The goal of the deep learning model 38 is to make a certain probabilistic prediction about the input through an artificial neural network type model and to adjust the model parameters so that this prediction is as similar as possible to the ground-truth probability. For example, if the actual value probability is y=(1.0, 0.0), but the deep learning model's prediction was Y=(0.4, 0.6), the model parameters should be adjusted so that Y has a value that is closer to y. Here, a criterion for judging ‘closeness’ or, in other words, a method for judging whether something is different is needed. In the field of deep learning, a loss function is defined for this purpose. The deep learning model 38 can use cross entropy as a loss function.

Referring again to FIG. 3 , the similar photo detection module 32 may search for similar photos in the photos using the subject, time, and location information of the photo according to the object identified by the object recognition module 26.

The photo management device 10 according to an embodiment has been described above with reference to FIGS. 1 to 7B. Hereinafter, a photo management method according to another embodiment will be described with reference to FIG. 8. However, the content described in the vision management device 10 that is equally applicable to the photo management method will be omitted or outlined.

Figure 8 is a flowchart of a photo management method according to another embodiment.

Referring to FIG. 8, the photo management method 44 according to another embodiment includes a step of storing photos (S46), analyzing images included in the stored photos to determine whether they are unnecessary photos, and if so, removing the photos. It includes a step of recommending deletion (S48), a step of listing photos recommended for deletion and outputting a display for selecting whether to delete the photos (S50).

Unnecessary photos may be at least one of shaken photos, dark photos, similar photos, and screenshots.

Figure 9 is a flowchart of steps for recommending deletion of the unnecessary picture in Figure 8.

If the photo is unnecessary, the step of recommending deletion of the photo (S48) includes a photo object recognition step (S48a) that identifies the object in the photo, a shaken photo detection step (S48b) that determines whether the photo is shaken, and a photo object recognition step (S48b) that determines whether the photo is shaken. It may include at least one of a dark photo detection step (S48c) for determining whether or not to use a dark photo, a similar photo detection step (S48d) for searching for similar photos in photos, and a screen shot search step (S48e).

The photo object recognition step (S48a) can identify objects in the photo by utilizing a deep learning model consisting of multiple convolution layers, a pooling layer, and a fully connected layer.

In the shaken photo detection step (S48b), when the blurriness of a photo is determined using Laplacian variance, the blurriness of the photo can be determined by converting the photo to black and white and then calculating the Laplacian variance.

The dark photo detection step (S48c) may determine whether the photo is dark by utilizing a deep learning model consisting of multiple convolution layers, a pooling layer, and a fully connected layer. The deep learning model used in the dark photo detection step (S48c) is trained by removing nodes in each layer with a certain probability when learning with training data, and when learning is completed and testing, all nodes are considered as original. And you can apply a dropout technique that multiplies the probability by the weight each node has.

In the similar photo detection step (S48d), similar photos can be searched among photos using the subject, time, and location information of the photo according to the object identified by the object recognition module.

In the screen shot search step (S48e), a screenshot can be searched among photos using whether or not the screenshot is included in the photo meta information table that is stored together when the photo is saved by the camera. In the case of a screenshot, the camera module checks whether the screenshot is taken and stores it in the screenshot field of the photo meta information table of the memory unit 16. Therefore, in the screenshot search step (S48e), the presence of a screenshot can be checked using the photo meta information table. This may be similar to using the photo meta information table in the similar photo detection step (S48d).

The photo management device and photo management method according to embodiments of the present invention have been described with reference to the drawings above. Below, specific embodiments and experimental examples will be described when the photo management device 10 is a photo management cloud server.

Example

1. Overview

In this embodiment, we mainly propose a cloud-based photo management system that provides the following functions. Cloud-based services can be said to be the most effective method due to advancements in communication. For example, we propose a cloud-based situational awareness system that solves the problems of server overload and increased latency using cloud-based methods.

First, it provides a function to distinguish between incorrectly taken photos. There is no need to keep poorly taken photos as dark and blurry photos. Create a model to classify dark photos using deep learning and classify blurred photos using OpenCV's photo analysis module.

Second, the cloud-based photo management system 52 provides a function to search for similar photos. People tend to take multiple photos of the same subject to get better photos. As a result, the number of similar photos is very large, making it difficult. To identify the subject of a photo, we adopt the deep learning-based YOLO object recognition model and perform a similarity test on the photo to find similar photos.

The cloud-based photo management system 52 provides the above two functions to check unnecessary photos and recommend photo deletion so that users can delete unnecessary photos themselves or automatically delete them. Saves storage space and supports effective photo management.

2. Structure of cloud-based photo management system

The proposed cloud-based photo management system 52 has the structure shown in FIG. 10. In FIG. 1, the cloud-based photo management system consists of a photo storage unit 54 corresponding to the memory unit and a photo analysis unit 56 corresponding to the control unit.

In the photo storage unit 54, the photo storage section manages photos using the Firebase database and provides a space to store the user's photos. The database stores meta information (latitude/longitude, date, time, file name, etc.) of photos taken with a smartphone and the location of the photos on the server. The database structure is shown in Figure 2.

As explained with reference to FIG. 2, the database used as the photo storage unit 54 includes a user table, a photo meta information table, a photo tag table, and a deletion information table. Includes (deletion information table).

The photo analysis unit 56 performs photo analysis to classify unnecessary photos, which is the main purpose of the proposed service. Unnecessary photos consist of at least one of the following: dark photos, blurry photos, similar photos, and screenshots.

The photo analysis unit 56 of the cloud-based photo management system 52 provides a function to automatically classify unnecessary photos. It manages storage space efficiently by deleting unnecessary photos and allows users to easily manage photos.

The Photo Object Recognition module uses the YOLO (You Only Look Once) algorithm to identify objects in photos. Photo tags are used for photo search, and the similar photo search function searches for similar photos using the photo's tags and meta information.

The shakey photo detection module is implemented using OpenCV's image analysis module and determines whether the photo is shaken.

The Dark photo Detection module is a deep neural network model created using Tensorflow and Keras to determine whether a given photo is dark.

The Similar photo search module searches for similar photos using tags and meta information of photos tagged by the Photo Object Recognition module.

The screen shot search module checks whether a screenshot is taken from the camera module and can search for a screenshot among photos using whether or not it is a screenshot included in the photo meta information table of the photo storage unit 54. .

3. Features of a cloud-based photo management system

The main functions and methods for providing the functions of the proposed cloud-based photo management system (52) are described.

First, the cloud-based photo management system 52 provides a photo storage function. This is a function to save photos on the user's smartphone and is a storage function like Naver Drive. A database was used as described above to manage the actual photo files.

Second, the cloud-based photo management system 52 provides a photo object tagging function using a photo object recognition module. This feature recognizes the subject of a photo and attaches appropriate tags. This is an addition to the smartphone's tagging feature and is used for photo search. The cloud-based photo management system 52 can tag photos as one of nine object types, including, but not limited to, people, animals, cars, furniture, books, bags, sports, food, and devices, as shown in Table 1. Types can be expanded, contracted, or transformed.

tag class person person animal Bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, teddy bear vehicle Bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, stop furniture Chair, sofa, bed, refrigerator, sink, toilet, dining table, clock book notes, books bag Handbags, backpacks, travel bags sports Frisbee, ski, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket food Wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, bottle Device Hair dryer, toaster, oven, microwave, cell phone, keyboard, remote control, mouse, laptop, TV, refrigerator

For object recognition, YOLO (You Only Look Once), a deep learning-based object recognition model, is used. The YOLO system was developed to detect all objects in real time using a single neural network. The YOLO system looks at an image once and predicts which object is where.

The YOLO system combines existing image recognition systems and methods for identifying the location of objects. The YOLO system defines the object detection problem as a regression problem that estimates the probability of spatially separated bounding boxes and attempts to solve it using a neural network. As a result, the YOLO system is composed of multiple convolution layers, pooling layers, and fully connected layers like in existing CNNs, as shown in FIG. 4.

In this example, the object recognition model was adjusted using pre-trained YOLOv3 weight values (yolov3.cfg available at https://pjreddie.com/media/files/yolov3.weights) and 1000 photos.

Third, the cloud-based photo management system 52 provides a function to distinguish shaken photos using a shakey photo detection module, and recognizes shaken photos using the OpenCV (Open Source Computer Vision) library.

The blurriness of the photo is assessed to determine whether the photo is blurry. As described above, to detect a blurred image, a method using a Gaussian basis filter, measurement of the degree of blurring of edges, analysis of edge sharpness, Laplacian dispersion, etc. can be used.

In this embodiment, Laplacian variance is adopted. Determine the blurriness of a photo, as the values can be easily calculated using the OpenCV library. If the variance value is less than the threshold (here alpha), the photo is determined to be blurry. The Laplacian measures what can be called the "curvature" or stress of a field. It tells how much the value of the field differs from the average value for surrounding points. Therefore, it can be used to measure the degree of shaking in a photo. As shown in Figure 5, to calculate the Laplacian variance, the photo is first converted to black and white.

The box below shows the algorithm that checks whether a photo is shaken. This algorithm uses the OpenCV library to convert a photo to black and white and calculate the variance of the Laplacian: cvtColor(photo, cv2.COLOR_BGR2GRAY) and Laplacian(photo, cv2. CV_64F).var(). The threshold (alpha) was set to 400 through experiments using sample photos.

Fourth, the cloud-based photo management system 52 provides a function to distinguish dark photos by utilizing a dark photo detection module. We use Tensorflow and Keras to create a deep learning model as shown in Figure 6 that distinguishes dark photos. Uses Tensorflow version 2.0 and Keras version 2.2.0.

Table 2 shows the structure of the model to which the general CNN structure is applied. The dark photo detection model structure is a typical CNN structure and has about 330K parameters. Because the number is relatively small, a lot of data and learning time are not needed.

Layer output shape number of parameters Convolution 64 x 64 x 32 896 Max Pooling 32 x 32 x 32 0 Convolution 32 x 32 x 32 9248 Max Pooling 16 x 16 x 32 0 Convolution 16 x 16 x 64 18496 Max Pooling 8 x 8 x 64 0 Dropout 8 x 8 x 64 0 Convolution 8 x 8 x 64 36928 Max Pooling 4 x 4 x 64 0 Dropout 4 x 4 x 64 0 Flatten 1024 0 High density 256 262400 Dropout 256 High Density 1257

Fifth, the cloud-based photo management system 52 provides a function to search similar photos using a dark photo search module. To get a good photo, we tend to take multiple photos of the same subject, so there are many similar photos. Similar photos are searched using the subject, time, and location information of the photo and notified to the user.

The box below shows the algorithm for searching similar photos. The algorithm allows three thresholds: alpha, beta and gamma. Alpha is the threshold for the time range, beta is for the longitude range, and gamma is for the latitude range. Photos whose time, latitude, and longitude are within the threshold are selected as similar photos.

Lastly, the cloud-based photo management system 52 provides a function to search for screen shots among photos using a screen shot search module. There are many cases where a screenshot of the screen is captured and saved as a photo to temporarily save the screen. Since these screenshots are unnecessary photos after serving the purpose for which they were saved, when taking a screenshot, whether it is a screenshot can be stored in the photo meta information table and then deleted at the same time when organizing unnecessary photos.

4. Experiment results

In this experimental example, we present the learning process and results of the dark photo detection model. The photo object recognition module adopts YOLOv3 as is.

The main purpose of the cloud-based photo management system 52 is to find and encourage unnecessary photos to be deleted. In this experimental example, the deletion recommendation function and results using the function provided by the cloud-based photo management system 52 are presented.

4.1 Dark photo detection model

Table 3 shows the environment for learning a dark photo detection model. The training and validation data include 354 light photos and 366 dark photos. The test data contains 46 bright photos and 34 dark photos. Early stopping is applied during the training process, and training is completed only at 13 epochs. As a binary classification problem, a model with good performance can be created with very few epochs.

item value Number of training data 612 Number of verification data 108 batch size 64 Epoch 100 Number of training parameters 328,225 loss function Binary cross entropy Optimizer Adam Optimizer activation function Convolution layer, 1st density layer: RELU Final density layer: Sigmoid number of test data 80

Figure 11 shows a model learning process according to an embodiment.

As can be seen in Figure 11, the prediction accuracy of the model measured using 80 test data is 95%, and this result supports that the generated model appropriately distinguishes dark photos.

4.2 Recommendations for deleting photos

The cloud-based photo management system 52 is a cloud-based photo management service that effectively manages photos and improves storage space efficiency. Therefore, the most important function is the deletion recommendation function. In other words, it is a good idea to check and delete unnecessary photos such as blurry photos, similar photos, and dark photos. When backing up photos from a smartphone to cloud storage, it goes through a photo analysis process, including determining the degree of shaking/darkness and searching for similar photos using photo object recognition, to determine whether or not to delete the photo.

Figure 12 is a partial image of the output unit 20 of a smart phone provided with a photo management service from the cloud-based photo management system 52.

In Figure 12(a), photos are classified according to the subject recognition results. Figure 12(b) is an image recommended for photo deletion.

Since it is divided into similar photos, blurry photos, and dark photos, you can see why it was recommended for deletion. Figure 12 (b) shows the appearance of the similar photo tab, and it can be seen that similar photos are appropriately recommended.

With reference to FIGS. 10 to 12, the cloud-based photo management system 52 that provides a cloud-based photo management service using deep learning has been described.

The above-described cloud-based photo management system 52 provides the following functions. First, it provides a function to back up photos stored on your smartphone. Second, it provides a function to check unnecessary photos to increase storage space efficiency. What differentiates the cloud-based photo management system 52 from other photo management services is its ability to check unnecessary photos. The cloud-based photo management system 52 can check unnecessary pictures such as dark photos, blurred photos, similar photos, and screenshots.

In the process of developing a cloud-based photo management system (52), we created a deep learning-based model using Tensorflow and Keras to classify dark photos. In addition, YOLOv3, a deep learning-based object recognition model, was applied to recognize the subject of the photo and use it to implement a similar photo search function. We also implemented a module to recognize shaken photos using OpenCV.

Through the example, we confirmed that dark photos, shaken photos, similar photos, and screenshots were properly searched. Currently, an application that can use the cloud-based photo management system 52 is provided on a smart phone, but the cloud-based photo management system 52 can be used through the web and various terminals. This is expected to contribute to providing convenience in photo management to many people.

Although the photo management device and photo management method according to embodiments of the present invention have been described with reference to the drawings, the present invention is not limited thereto.

The photo management device 10 described above may be implemented by a computing device including at least some of a processor, memory, user input device, and presentation device. Memory is a medium that stores computer-readable software, applications, program modules, routines, instructions, and/or data that are coded to perform specific tasks when executed by a processor. The processor may read and execute computer-readable software, applications, program modules, routines, instructions, and/or data stored in memory. A user input device may be a means for allowing a user to input a command that causes the processor to execute a specific task or to input data required to execute a specific task. User input devices may include a physical or virtual keyboard, keypad, key buttons, mouse, joystick, trackball, touch-sensitive input means, or microphone. Presentation devices may include displays, printers, speakers, or vibrating devices.

Computing devices may include a variety of devices such as smartphones, tablets, laptops, desktops, servers, and clients. A computing device may be a single stand-alone device or may include multiple computing devices operating in a distributed environment comprised of multiple computing devices cooperating with each other through a communication network.

In addition, the photo management method described above includes a processor and computer readable software, applications, program modules, routines, instructions, and/or data structures, etc. coded to perform the photo management method when executed by the processor. It can be executed by a computing device having a stored memory.

The above-described embodiments can be implemented through various means. For example, the present embodiments may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the photo management method according to the present embodiments uses one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), and PLDs (Programmable Logic Devices). , can be implemented by FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, or microprocessors.

For example, a photo management method using a deep learning model according to embodiments can be implemented using an artificial intelligence semiconductor device in which neurons and synapses of a deep neural network are implemented with semiconductor devices. At this time, the semiconductor device may be currently used semiconductor devices, such as SRAM, DRAM, or NAND, or may be next-generation semiconductor devices such as RRAM, STT MRAM, or PRAM, or a combination thereof.

When implementing the photo management method according to the embodiments using an artificial intelligence semiconductor device, the results (weights) of learning the deep learning model with software can be transferred to synapse-mimicking elements arranged in an array or learning can be performed on the artificial intelligence semiconductor device. It may be possible.

In the case of implementation using firmware or software, the photo management method according to the present embodiments may be implemented in the form of a device, procedure, or function that performs the functions or operations described above. Software code can be stored in a memory unit and run by a processor. The memory unit is located inside or outside the processor and can exchange data with the processor through various known means.

Additionally, terms such as "system", "processor", "controller", "component", "module", "interface", "model", or "unit" described above generally refer to computer-related entities hardware, hardware and software. It may refer to a combination of, software, or running software. By way of example, but not limited to, the foregoing components may be a process, processor, controller, control processor, object, thread of execution, program, and/or computer run by a processor. For example, both an application running on a controller or processor and the controller or processor can be a component. One or more components may reside within a process and/or thread of execution, and the components may be located on a single device (e.g., system, computing device, etc.) or distributed across two or more devices.

The above description is merely an illustrative explanation of the technical idea of the present disclosure, and those skilled in the art will be able to make various modifications and variations without departing from the essential characteristics of the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of the present disclosure, but rather to explain it, so the scope of the present technical idea is not limited by these embodiments. The scope of protection of this disclosure should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this disclosure.

Claims (15)

input unit;
A communication unit that exchanges data with other devices through a communication network;
a memory unit that stores photos;
a control unit that analyzes images included in the stored photos to determine whether they are unnecessary photos, and recommends deletion of the photos if they are unnecessary photos; and
An output unit that lists photos recommended for deletion by the control unit and outputs a display for selecting whether to delete the photos,
Unnecessary photos are at least one of blurry photos, dark photos, similar photos, and screenshots.
The control unit includes a photo object recognition module for identifying objects in the photo, a shaken photo detection module for determining whether the photo is shaken, a dark photo detection module for determining whether the photo is dark, and similar photos in the photos. At least one of a similar photo detection module for searching, a screenshot search module for searching for screenshots among the photos,
The photographic object recognition module divides the image included in the photograph into several grids, each grid predicts a certain number of bounding boxes to identify the location of the object, and a confidence score is assigned to each bounding box, Identify objects in the photo using each predicted bounding box and confidence score,
The shaken photo detection module determines whether the photo is blurred by using a method using a Gaussian basis filter, a method of measuring the degree of blur at edges, an edge sharpness analysis method, and Laplacian variance, and determines whether the photo is shaken. When the detection module determines the blurriness of the photo using Laplacian variance, converts the photo to black and white and calculates Laplacian variance to determine the blurriness of the photo,
The similar photo detection module searches for similar photos in the photos using the subject, time, and location information of the photo according to the object identified by the object recognition module,
The screenshot search module searches for a screenshot among photos using whether or not the screenshot is included in a photo meta information table stored together when saving the photo.
The dark photo detection module determines whether the photo is dark by utilizing a deep learning model consisting of multiple convolution layers, a pooling layer, and a fully connected layer,
When learning with training data, the deep learning model trains the nodes in each layer by removing them with a certain probability, and when learning is completed and testing, it considers all nodes as they were and sets the weight each node has to the probability. Photo management device with multiplied dropout technology. delete delete According to paragraph 1,
The photo object recognition module is a photo management device that identifies objects in the photo by utilizing a deep learning model consisting of multiple convolution layers, a pooling layer, and a fully connected layer. delete delete delete delete According to paragraph 1,
The photo management device is a cloud server or a communication terminal. saving photos;
Analyzing the images included in the stored photos to determine whether they are unnecessary photos, and recommending deletion of the photos if they are unnecessary photos; and
A step of listing the photos recommended for deletion and outputting a display for selecting whether to delete the photos,
The unnecessary photo is at least one of a shaken photo, a dark photo, a similar photo, and a screenshot,
If the photo is an unnecessary photo, the step of recommending deletion of the photo includes a photo object recognition step of identifying an object in the photo, a shaken photo detection step of determining whether the photo is shaken, and a dark photo detection step of determining whether the photo is dark. It includes at least one of a photo detection step, a similar photo detection step of searching for similar photos in the photos, and a screenshot search step of searching for screenshots in the photos,
The photographic object recognition step divides the image included in the photograph into several grids, each grid predicts a certain number of bounding boxes to identify the location of the object, and a confidence score is assigned to each bounding box, Identify objects in the photo using each predicted bounding box and confidence score,
The shaken photo detection step determines whether the photo is shaken by determining the blurriness of the photo using a method using a Gaussian basis filter, a method of measuring the degree of blur at edges, an edge sharpness analysis method, and Laplacian variance, and determining whether the photo is shaken. When the detection step determines the blurriness of the photo using Laplacian variance, convert the photo to black and white and calculate the Laplacian variance to determine the blurriness of the photo,
The similar photo detection step searches for similar photos in the photos using the subject, time, and location information of the photo according to the object identified by the object recognition step,
The screenshot search step is to search for a screenshot among photos using whether or not the screenshot is included in a photo meta information table that is stored together when saving the photo.
The dark photo detection step determines whether the photo is dark using a deep learning model consisting of multiple convolution layers, a pooling layer, and a fully connected layer,
When learning with training data, the deep learning model trains the nodes in each layer by removing them with a certain probability, and when learning is completed and testing, it considers all nodes as they were and sets the weight each node has to the probability. Photo management method using multiplied dropout technology. delete According to clause 10,
The photo object recognition step is photo management that identifies objects in the photo using a deep learning model consisting of multiple convolution layers, a pooling layer, and a fully connected layer. method.
delete delete delete

Images