KR20210057428A - Method, user device and recording medium for computer program - Google Patents

Abstract

Provided are an image analysis method, a user device performing the image analysis method, and a computer program executing the user device and the image analysis method. The image analysis method of the present invention comprises the steps of: determining whether to analyze one or more images stored in a memory; selecting an analysis target image from the one or more images; analyzing the analysis target image; and outputting the analysis result.

Description

Video analysis method, user device and computer program {METHOD, USER DEVICE AND RECORDING MEDIUM FOR COMPUTER PROGRAM}

The present invention relates to an image analysis method, a user device, and a computer program for analyzing an image and outputting the analyzed result. Specifically, the present invention analyzes an image to identify a highlight section, outputs information on at least one highlight section, or outputs a highlight image composed of at least one highlight section, thereby making an image meaningful for output. It relates to a method capable of providing a user with a user device, a user device, and a program stored in a computer-readable recording medium capable of executing the method.

In recent years, the trend of communication through instant messenger applications and SNS (Social Network Service) is changing to video-oriented communication beyond the level of simple conversation. Users can use the video after editing the video for output, rather than using the video taken personally or obtained from the outside directly for communication. For example, a user can create a meaningful output image by removing portions of the acquired video that are not suitable for output (preparing to shoot in front of the image, finishing shooting behind the image, severely shaken or incorrectly photographed, etc.). . For this reason, most services that support video-oriented communication provide basic editing functions for editing videos.

A user who edits a video generates an output video of an appropriate length by selecting a necessary part excluding an unnecessary section while repeatedly playing the video. However, as communication using moving pictures increases, there is a limit to the user's direct editing of all moving pictures. Therefore, there is a need for a method of automatically analyzing the moving pictures stored in the memory and providing the analyzed results to the user. In addition, since editing of a moving picture requires a considerable amount of device resources, care must be taken in selecting a timing for automatically analyzing the moving picture stored in the memory.

An object of the present invention is to provide an image analysis method for analyzing an image and outputting the analyzed result.

In addition, the present invention analyzes a frame constituting an image, identifies a highlight section constituting the image, and then outputs information about a trim section based on the characteristics of the identified highlight section, or information about the highlight image. It is an object of the present invention to provide an image analysis method that outputs.

Another object of the present invention is to provide a user device as an image analysis apparatus for performing the image analysis method according to the present invention.

An object of the present invention is to provide a computer program stored in a computer-readable recording medium for executing an image analysis method according to the present invention.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are those of ordinary skill in the technical field to which the present disclosure pertains from the following description (hereinafter, "common technician It will be clearly understandable to "").

According to an aspect of the present invention, an image analysis method performed by a user device including a memory and at least one processor includes determining whether to analyze one or more images stored in the memory, the one or more images Selecting an analysis target image, analyzing the analysis target image, and outputting an analysis result of at least one of the analysis target images.

In the image analysis method according to the present invention, the step of determining whether to perform the image analysis may be performed based on whether the user device is in an idle state.

In the image analysis method according to the present invention, the step of determining whether to perform the image analysis may be performed based on a usage pattern of the user device.

In the image analysis method according to the present invention, the step of determining whether to perform the image analysis comprises: the user device is in an idle state, and the user device is not used for a predetermined time based on a usage pattern of the user device. If it is determined not to be, it may be determined to analyze the image.

In the image analysis method according to the present invention, in the selecting of the analysis target image, an image having the latest generation date among one or more images stored in the memory may be selected as the analysis target image.

In the image analysis method according to the present invention, the analyzing the analysis target image comprises: analyzing each of a plurality of frames constituting the analysis target image, and at least one image identified based on the analysis result of the frame. It may include classifying a section into a highlight section and a non-highlight section, and analyzing each of the image sections classified as the highlight section.

In the image analysis method according to the present invention, the analyzing of the frame includes whether at least one of a brightness value and a color value of pixels constituting each of the plurality of frames, and an amount of an edge in the frame meets a predetermined criterion. By determining whether or not, it is possible to identify a frame that is not to be output.

In the image analysis method according to the present invention, the step of classifying the video section includes: when the length of the video section is greater than or equal to a predetermined length, the video section is classified as a highlight section, and when the length of the video section is less than a predetermined length , The video section can be classified as a non-highlight section.

In the video analysis method according to the present invention, the analyzing of the video section includes at least one of a result of analysis of each of a plurality of frames constituting the video section, a difference value between frames, and a length of the video section. It can be calculated as a characteristic of a section.

In the image analysis method according to the present invention, the outputting of the analysis result may output information on at least one highlight section based on a characteristic of the image section.

In the image analysis method according to the present invention, the outputting of the analysis result may include outputting an analysis result for the corresponding image when an edit screen request for at least one of the analysis target images is received from a user of the user device. have.

In the image analysis method according to the present invention, the outputting of the analysis result comprises: receiving a selection of one or more images stored in the memory from a user of the user device, and requesting an edit screen for the selected image Is received, determining whether the selected image corresponds to the analysis target image and the analysis result for the selected image is stored in the memory, and when it is determined that the analysis result for the selected image is stored, the It may include the step of outputting the analysis result.

In the image analysis method according to the present invention, when it is determined that the analysis result for the selected image is stored, an edit UI element for the selected image may be displayed together with the analysis result.

According to another aspect of the present invention, a user device performing an image analysis method includes a memory for storing one or more images, and at least one processor, and the processor performs analysis of one or more images stored in the memory. Determining whether to do so, selecting an analysis target image from among the one or more images, analyzing the analysis target image, and outputting the analysis result may be performed.

In the user device according to the present invention, the processor analyzes the image when it is determined that the user device is in an idle state and the user device is not used for a predetermined time based on a usage pattern of the user device. You can decide to do it.

In the user device according to the present invention, the processor, in the step of analyzing the analysis target image, analyzing each of a plurality of frames constituting the analysis target image, identified based on an analysis result of the frame Classifying at least one image section into a highlight section and a non-highlight section, and analyzing each of the image sections classified as the highlight section may be performed.

In the user device according to the present invention, in the step of analyzing the video section, the processor comprises at least one of an analysis result of each of a plurality of frames constituting the video section, a difference value between frames, and a length of the video section. In the step of calculating one as a characteristic of a corresponding video section and outputting the analysis result, information on at least one highlight section may be output based on the characteristic of the video section.

In the user device according to the present invention, in the step of outputting the analysis result, when a request for an edit screen for at least one of the analysis target images is received from a user of the user device, the processor receives the analysis result for the corresponding image. Can be printed.

In the user device according to the present invention, the processor, in the step of outputting the analysis result, receiving a selection of one or more images stored in the memory from a user of the user device, with respect to the selected image When an edit screen request is received, determining whether the selected image corresponds to the analysis target image and an analysis result for the selected image is stored in the memory, and determining that the analysis result for the selected image is stored In this case, the step of outputting the analysis result may be performed.

A program stored in a computer-readable recording medium according to another aspect of the present invention may be a computer program that executes the image analysis method according to the present invention.

Features briefly summarized above with respect to the present disclosure are only exemplary aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

According to the present invention, an image analysis method for analyzing an image and outputting the analyzed result may be provided.

In addition, according to the present invention, after analyzing a frame constituting an image, identifying a highlight section constituting the image, outputting information about a trim section or outputting information about a highlight image based on the characteristics of the identified highlight section. An output image analysis method may be provided.

In addition, according to the present invention, a user device as an image analysis apparatus for performing the image analysis method according to the present invention may be provided.

In addition, a computer program stored in a computer-readable recording medium for executing the image analysis method according to the present invention may be provided.

The effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram showing a system in which an image analysis method according to an embodiment of the present invention can be used.
2 is a block diagram illustrating a user device 200 that performs an image analysis method according to an embodiment of the present invention.
3 is a diagram for explaining an image analysis method according to an embodiment of the present invention.
4 is a diagram for explaining an embodiment of image analysis according to the present invention.
5 is a diagram illustrating a plurality of frames constituting an image to be analyzed.
6 is a diagram illustrating a plurality of image sections acquired from an image to be analyzed after frame analysis.
7 is a diagram for explaining a method of selecting a trim section from an image.
8 is a diagram for explaining an embodiment of providing a highlight image generated by performing the image analysis method of the present invention.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

In describing an embodiment of the present disclosure, when it is determined that a detailed description of a known configuration or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, parts not related to the description of the present disclosure in the drawings are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, when a component is said to be "connected", "coupled" or "connected" with another component, it is not only a direct connection relationship, but also an indirect connection relationship in which another component exists in the middle. It can also include. In addition, when a certain component "includes" or "have" another component, it means that other components may be further included rather than excluding other components unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise noted. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment is referred to as a first component in another embodiment. It can also be called.

In the present disclosure, components that are distinguished from each other are intended to clearly describe each feature, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated into one hardware or software unit, or one component may be distributed to form a plurality of hardware or software units. Therefore, even if not stated otherwise, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment consisting of a subset of components described in an embodiment is also included in the scope of the present disclosure. In addition, embodiments including other elements in addition to the elements described in the various embodiments are included in the scope of the present disclosure.

In addition, in the present specification, the network may be a concept including all wired and wireless networks. In this case, the network may mean a communication network in which data exchange between a device, a system, and a device can be performed, and is not limited to a specific network.

In addition, in the present specification, the device may be a mobile device such as a smart phone, a tablet PC, a wearable device, and a head mounted display (HMD), as well as a fixed device such as a PC or a home appliance having a display function. In addition, as an example, the device may be a computing device capable of operating as a server, a vehicle, or an Internet of Things (IoT) device. That is, in the present specification, a device may refer to devices capable of performing the image analysis method according to the present invention, and is not limited to a specific type.

In addition, in the present specification, "image" may include all kinds of information that can be visually recognized by a user through a display provided in the device, such as a moving image and a streaming image, as well as a still image.

System and device configuration

1 is a diagram showing a system in which an image analysis method according to an embodiment of the present invention can be used.

The system according to the present invention may include a server 110 and one or more user devices 101, 102, 103 connected via a network 104.

Each of the user devices 101, 102, and 103 may be referred to as a client or a user terminal, and may access the server 110 through the network 104 to download and output a desired image or content. In addition, each of the user devices 101, 102, and 103 may acquire and store an image using a photographing means (eg, a camera module provided in a smartphone). The user device may perform an image analysis method according to an embodiment of the present invention on an image downloaded through the network 104 or acquired through a photographing means.

The server 110 may store a vast amount of images and contents in a storage space in the server 110 or in a separate database. In addition, the server 110 may identify a user, and may accumulate and store various information such as information about the user, information about images and contents.

In addition, the server 110 may provide a service such as an instant messenger application and a social network service (SNS). A user using the service may access the server 110 providing the service by inputting predetermined access information (ID and password) through a user device. The server 110 may identify a user who has accessed through the access information received from the user device.

Using the service, the user may transmit an image stored in the user device to another user device connected to the network 104 or upload it to a storage space in the server 110. In this case, the transmitted or uploaded image may be an image on which the image analysis method according to an embodiment of the present invention has been performed.

2 is a block diagram illustrating a user device 200 that performs an image analysis method according to an embodiment of the present invention.

Referring to FIG. 2, the user device 200 may include a processor 210, a memory 220, a transmission/reception unit 230, an input unit 240, a photographing unit 250, and an output unit 260. The user device 200 may further include other components related to the operation and function of the device, and is not limited to the above-described embodiment.

The processor 210 may control operations of other components in the user device 200. For example, the processor 210 may process information acquired through the input unit 240, the photographing unit 250, and the transmission/reception unit 230. Also, the processor 210 may read and process information stored in the memory 220. The processor 210 may output the processed information through the output unit 260, store it in the memory 220, or transmit the processed information to the outside through the transmission/reception unit 230.

For example, the processor 210 may perform an image analysis method according to the present invention on an image captured by the photographing unit 250 or an image stored in the memory 220. After performing the image analysis method according to the present invention, the processor 210 stores information on the image for which the image analysis has been completed in the memory 220, outputs through the output unit 260, or through the transmission/reception unit 230. Can be transmitted outside.

The memory 220 is obtained from the outside of the user device, such as information received from the server 110 or other user device, information obtained through the input unit 240 of the user device, and information obtained from the photographing unit 250 of the user device. One can store information. Also, the memory 220 may store information generated inside the user device 200. For example, an image or information about an image generated after the image analysis method according to the present invention is performed may be stored in the memory 220. For example, the memory 220 may include a database.

The transmission/reception unit 230 may exchange data with the server 110 or other user devices through a network. The transmission/reception unit 230 may include all types of wired/wireless communication modules capable of performing communication with the outside.

The input unit 240 may obtain input information through various sensors included in the user device 200. The user device 200 includes various sensors such as, for example, a pressure sensor, a capacitive touch sensor, an acceleration sensor, a gyro sensor, a GPS (Globap Positioning System), an acoustic sensor (microphone), and a photosensitive sensor (illumination sensor). A mechanical button or the like may be provided. The user device 200 may obtain information sensed by the sensor or an input of a mechanical button as input information. The input information may include user input information obtained by directly manipulating the user device 200 by the user. In addition, the input information may include device state information automatically detected by a state of the user device 200 or an external situation.

The photographing unit 250 may include any kind of photographing means capable of obtaining a still image or a moving picture. For example, the photographing unit 250 may be a camera module provided in a smartphone.

The output unit 260 may externally output information acquired or received by the user device 200, information processed by the user device 200, and the like. The output unit 260 may include, for example, a display outputting visual information, a speaker outputting sound information, a vibrator outputting tactile information, and the like.

Video analysis method

3 is a diagram for explaining an image analysis method according to an embodiment of the present invention.

An image analysis method according to an embodiment of the present invention includes determining whether to perform image analysis (S310), selecting an image to be analyzed (S320), performing image analysis (S330), and outputting an image analysis result (S340). I can.

The processor 210 of the user device may determine whether to perform image analysis according to an embodiment of the present invention (S310). Whether to perform image analysis may be determined based on a state of the user device, a usage pattern of the user device, and/or a predetermined condition.

As described above, a significant amount of resources (memory, computing time, battery power, etc.) are required to perform the image analysis method according to an embodiment of the present invention. In order to properly allocate and use resources, for example, image analysis may be performed only when the user device is in an idle state.

Whether the user device is in an idle state may be determined by various methods. For example, it may be determined whether the user device is in an idle state based on information detected by various sensors included in the user device. For example, if the display is in an off state, it may be determined that the display is in an idle state. Alternatively, whether the user device is in a stationary state or in motion may be determined based on detection information of an acceleration sensor and/or a gyro sensor, and if the user device is in a stationary state, it may be determined that it is in an idle state. Alternatively, it may be determined whether the user device is in use based on the pressure sensor of the user device, detection information of the capacitive touch sensor, and button input information provided in the user device. For example, if there is no user input information, it may be determined that the user device is in an idle state. For example, when the user device is in a screen lock state, it may be determined that the user device is in an idle state. Alternatively, the idle state of the user device may be determined based on the detection information of the photosensitive sensor. For example, when the intensity of light detected by the photosensitive sensor is less than or equal to a threshold, it may be determined that the user device is in an idle state.

Also, whether to perform image analysis according to an embodiment of the present invention may be determined based on a usage pattern of a user device. In order to perform the image analysis method of the present invention, an analysis time of more than a predetermined time is required. Accordingly, image analysis may be performed only when an analysis time of more than a predetermined time can be secured by analyzing the usage pattern of the user device.

User device usage patterns can be continuously monitored and recorded. The processor can accumulate monitoring information on the usage pattern of the user device by continuously monitoring information detected by various sensors provided in the user device, button input, usage time of the user device, and usage (function) of the user device. . The processor may determine whether a minimum time required for image analysis can be secured by performing prediction according to a random variable based on the accumulated monitoring information.

For example, assuming a usage pattern in which a user device is manipulated several times every hour excluding bedtime from midnight to 7 a.m., based on monitoring information on the usage pattern of the user device, from midnight to 7 a.m. Losing can be predicted with a high probability that the corresponding user device will neither move nor be used. In addition, the probability of not being used in other times can be predicted very low. The probability that the user device will not be used may be predicted in units of 1 minute, or may be predicted in units of a predetermined time, such as in units of minutes, 1 hour, or several hours.

As a modified example, as a result of monitoring the usage pattern of the user device, it is determined that the resource required for image analysis can be secured considering the purpose of using the user device during the time zone, even if the user device is used in a high probability. If so, the corresponding time zone may be predicted as a time zone in which image analysis can be performed.

For example, as a result of monitoring the usage time and usage of the user device, assuming a usage pattern that uses only simple functions that do not require a lot of resources from 10 pm to 11 pm, the time zone is a resource required for video analysis. It is determined that is sufficient, and it can be predicted that image analysis can be performed.

The processor may determine whether to perform the image analysis according to the present invention by performing prediction based on information on monitoring the usage pattern of the user device.

Whether to perform image analysis according to an embodiment of the present invention may be determined by determining whether a prediction according to a state of a user device and/or a usage pattern of a user device meets a predetermined condition.

The predetermined condition may be set in advance or may be adaptively set based on a resource of a user device performing image analysis according to the present invention. Or, it can be set arbitrarily by the user. The predetermined condition for determining that the user device performs the image analysis according to the present invention may be, for example, a condition in which the user device is in an idle state and is not used for a predetermined time (eg, 2 hours).

When detecting that the user device is in an idle state, the processor may predict a probability that the user device will not be used for a predetermined period of time based on a usage pattern of the user device. When the probability is greater than or equal to a predetermined threshold, the processor may determine that the predetermined condition is satisfied and determine that the image analysis according to the present invention is performed.

As a modified example, when it is determined that resources necessary for image analysis according to the present invention can be secured for a predetermined time based on the usage pattern of the user device, the processor may determine to perform image analysis according to the present invention.

As another modification, the processor may determine whether to perform image analysis by considering a probability that the user device will not be used for a predetermined period of time, whether resources can be secured for a predetermined period of time, and the like.

In addition, the remaining battery power of the user device, the remaining memory capacity, whether the battery is being charged, whether or not a network (eg, WiFi) is connected may be additionally considered.

For example, when the remaining battery power is at least a predetermined threshold, the remaining memory capacity is at least a predetermined threshold, the battery is being charged, and when it corresponds to at least one of a case connected to the network, according to the present invention. It can be decided to perform image analysis.

As another modification, whether or not to perform image analysis may be determined by the user's manipulation, regardless of the state and usage pattern of the user device. For example, it may be determined that the image analysis according to the present invention is performed regardless of the state and usage pattern of the user device by manually instructing the user to perform the image analysis by selecting the image stored in the memory or the image acquired through the photographing unit. May be.

If it is determined in step S310 to perform image analysis according to the present invention, the processor 210 may select an image to be analyzed (S320). The analysis target image may be an image acquired by a photographing unit. Alternatively, the analysis target image may be selected from one or more images stored in the memory by a user's manipulation. Alternatively, the image to be analyzed may be selected by the processor according to a predetermined selection criterion. For example, the processor may select an image to be analyzed recently or stored among one or more images stored in the memory. In this case, the selected image may be an image for which image analysis according to the present invention has not been performed. If the recently photographed or stored image is an image that has already been analyzed according to the present invention, the next recently photographed or stored image may be selected as the analysis target image. That is, the most recently photographed or stored image among images other than the image that has already been analyzed may be selected as the image to be analyzed.

When the image is captured, when it is stored, and/or whether the image is analyzed as metadata of the image, it may be stored together with the image in a memory of the user device. In order to select an image to be analyzed, the processor may check the capturing time, storage time, and/or analysis of each image using the metadata.

When the analysis target image is selected in step S320, the processor 210 may perform image analysis according to the present invention on the selected analysis target image (S330).

4 is a diagram for explaining an embodiment of image analysis according to the present invention.

The image analysis method shown in FIG. 4 may be an embodiment of the image analysis in step S330 of FIG. 3.

The analysis target image may be a moving image including one or more still images. The moving picture represents the motion of an object in the still image by displaying a plurality of still images, such as 30 or 60 images per second according to a predetermined frame rate. Still images constituting a moving picture may be referred to as a field in interlaced scanning, and may be referred to as a frame in progressive scanning. However, in this specification, a still image constituting a moving picture is collectively referred to as a "frame."

Since a video consists of dozens of still images per second, the amount of data can be vast. Accordingly, moving pictures can be efficiently compressed, stored, and transmitted using various video codec technologies. The video encoder may encode a video and output a bitstream. The bitstream may include compression information capable of reconstructing each of a plurality of frames constituting a video. The video decoder may reconstruct each frame constituting the moving picture by receiving the bitstream and decoding the bitstream, and then display it through the output unit.

When the analysis target image selected by the processor is encoded data of a moving picture such as a bitstream, the processor may decode the encoded data to restore each frame constituting the analysis target image, and then perform image analysis according to the present invention.

5 is a diagram illustrating a plurality of frames constituting an image to be analyzed.

As shown in FIG. 5, the image to be analyzed may include a plurality of frames that are continuously displayed. The number at the bottom of each frame is an identifier for identifying the frame.

Referring back to FIG. 4, in order to perform image analysis according to the present invention, the processor may analyze each frame constituting the analysis target image (S410).

The analysis of the frame in step S410 is a step for identifying and/or removing frames that are not suitable for output.

For example, a frame that is too bright or too dark may be identified and/or removed in the frame analysis step of S410. For this, brightness values of pixels constituting the frame may be used. For example, it is possible to determine whether the frame is too bright or too dark by determining a representative brightness value of a frame by using the brightness values of pixels constituting the frame and comparing the representative brightness value with a predetermined threshold. The representative brightness value may be calculated using one or more statistical values such as an average value, a median value, a variance, and a standard deviation of the brightness values of pixels constituting a frame.

Alternatively, for example, a frame including only one color may be identified and/or removed in the frame analysis step of S410. For this, color values of pixels constituting the frame may be used. For example, when a color value of all pixels constituting a frame or a predetermined number of pixels or more is included in a range of a predetermined color value, it may be determined that the frame includes only one color.

Alternatively, a frame in which shaking is severe enough to obscure the outline of the object included in the frame may be identified and/or removed. For this, various edge extraction algorithms or edge histograms may be used. When the amount of edges extracted using an edge extraction algorithm or an edge histogram is less than or equal to a predetermined threshold, the corresponding frame may be determined as a frame with severe shaking.

The above-described frame analysis method may be applied by combining one or two or more, and other various criteria and methods for identifying frames that are not suitable for display may be applied.

In the above, values as a reference for frame analysis, such as a predetermined threshold, a predetermined number, and a predetermined color value range, may be set in advance or may be set by a user. Alternatively, it may be adaptively set based on the characteristics of the frame. For example, for a frame having a low brightness value, a predetermined threshold compared to the amount of the edge may be set to be high.

When a frame that is not appropriate for an image is identified in step S410, a plurality of image sections may be obtained by excluding the identified frame from an image to be analyzed.

6 is a diagram illustrating a plurality of image sections acquired from an image to be analyzed after frame analysis.

For example, if Frame 1, Frame 4, Frame 8, and Frame 9 are identified as inappropriate output frames in FIG. 5, these frames are excluded from the image to be analyzed, as shown in FIG. Sections 1 to 3) may be obtained.

When a plurality of image segments are acquired, the processor may classify each image segment (S420).

In step S420, each of the acquired image sections is classified into a highlight section and a non-highlight section. Here, the highlight section means an image section expected to be necessary for editing the image, and the non-highlight section may mean an image section that is not necessary for editing the image or has no visual significance.

For the classification of the video section in step S420, the length of each video section may be used. For example, an image section less than a predetermined reference length may be determined as an imagely meaningless section and classified as a non-highlight section. The predetermined reference length may be preset, set by a user, or adaptively set based on characteristics of an image to be analyzed. For example, if it is determined that the total length of the analysis target image is greater than or equal to a predetermined length, the predetermined reference length may be set to a relatively large value.

The predetermined reference length may be expressed by a predetermined time or by a predetermined number of frames. For example, an image section of 0.5 seconds or less or an image section composed of a predetermined number of frames or less may be classified as a non-highlight section.

In the embodiment shown in FIG. 6, assuming that the predetermined number is 2, an image section 1 consisting of two frames is classified as a non-highlight section, and an image section 2 and an image section 3 consisting of 3 and 4 respectively It can be classified as a highlight section.

For an image section classified as a highlight section, a key frame may be extracted. The key frame may mean a frame that is optimally imaged within an image section. Key frames can be extracted through frame analysis. For example, the key frame may be extracted based on the sharpness of the screen and the diversity of colors within the frame. For example, a frame having both high clarity and color diversity may be extracted as a key frame. Alternatively, analysis results of various indicators analyzed in the frame analysis step of step S410 may be used together.

The key frame is a frame representing a corresponding video section and may be extracted and stored as a still image. The key frame may be used as a thumbnail image for a corresponding video section.

The processor may perform image section analysis of step S430 for each of the highlight sections classified in step S420.

The video section analysis in step S430 is a step for calculating the characteristics of the video section. The characteristics of the image section thus calculated may be used in the step of outputting an image analysis result (S340), which will be described later.

Through the analysis of the image section in step S430, section characteristics such as which colors are distributed in the corresponding highlight section, whether the screen composition is complex or simple, and whether there is a lot of movement or less may be calculated. In order to calculate the characteristics of the highlight section, an analysis result of each frame included in the highlight section may be used.

For example, what colors are distributed in the highlight section, whether colors are varied, etc. may be calculated using a color histogram for each frame included in the highlight section. Also, whether the screen composition is complex or simple, or whether many objects are included can be calculated using an edge histogram.

Also, whether there is a lot of motion or not may be calculated using a frame-by-frame difference between frames included in the highlight section. Since a highlight section can include a large number of frames, there can be multiple differences between frames, and in this case, one or more statistical values such as average, median, variance, and standard deviation of the multiple difference values are used. Thus, the characteristics of the highlight section can be calculated.

The characteristics of the video section are not limited to the above-described contents, and all information (eg, length, resolution, brightness, color, etc.) about the video section may be used as the section property.

Referring back to FIG. 3, after the image analysis in step S330 is performed, the processor may output the image analysis result in step S340. As described above, the characteristics of the image section calculated in step S430 may be used in step S340.

The output of the image analysis result in step S340 may be the output of information on at least one highlight section. In this case, the output of step S340 may be an output of at least one highlight section through the display. Alternatively, the output of step S340 may be to store information on at least one highlight section in the memory 220. In this case, the information on the at least one highlight section may be stored in a memory of the user device as metadata of the corresponding image together with an identifier (ID) of the corresponding image. The at least one highlight section to be output in step S340 may be a trim section or a highlight image.

The output of the information on the trim section or the output of the highlight image information may be performed when an edit screen request is input for an image designated by a user. For example, when an image to be uploaded on an SNS is selected by the user and an edit screen request UI of the corresponding image is selected, information about a trim section or highlight image information of the selected image may be output.

When information about the trim section is output as a result of image analysis, one video section selected from one or more highlight sections may be output as a trim section. The selected trim section may be output by being displayed on the display so that the user can identify it. Alternatively, information on the selected trim section may be stored in the memory.

The information on the trim section is, for example, information that can specify the trim section, and may be information indicating the start and end of the trim section, or information indicating the start and duration of the trim section. have. In this case, the information may be expressed as the number of frames or time. Alternatively, an image corresponding to the trim section may be separately stored in the memory.

After the image analysis according to the present invention is performed, the user can immediately receive the trim section of the corresponding image through the display. In addition, when the analysis of the image is completed and information on the trim section of the corresponding video is stored in the memory, if there is a request from the user, the trim section of the video may be immediately provided to the user without further analysis.

A criterion for selecting a trim section from a plurality of highlight sections may be set in advance or may be set by a user. For example, a section having the longest length among a plurality of highlight sections may be selected as the trim section. To this end, a length may be used as a characteristic of each highlight section. After comparing the lengths of each of the plurality of highlight sections, the processor may select the longest section as the trim section. For example, in the example shown in FIG. 6, the trim section may be selected as the image section 3.

For example, a section having a length closest to a predetermined length among a plurality of highlight sections may be selected as the trim section. The predetermined length may be set in advance or may be set by a user. For example, the predetermined length may be set by restrictions of an application for transmitting/receiving or uploading a video.

Alternatively, for example, a section with the most or the least amount of motion in the screen among the plurality of highlight sections may be selected as the trim section. To this end, as a characteristic of each highlight section, a characteristic calculated using, for example, a frame-by-frame difference between frames may be used.

Alternatively, for example, a section having the highest or lowest color diversity among the plurality of highlight sections may be selected as the trim section. To this end, as a characteristic of each highlight section, a characteristic calculated using, for example, a color histogram may be used.

Alternatively, for example, the clearest section among the plurality of highlight sections may be selected as the trim section. To this end, as a characteristic of each highlight section, for example, a characteristic calculated using an edge histogram may be used.

Various criteria for selecting the trim section may be applied alone or in combination with other criteria. When two or more criteria are combined and applied, different weights are assigned to each criterion, so that the priorities between the criteria can be different.

One video section may be provided as a trim section, or two or more video sections may be provided. When two or more video sections are provided as trim sections, some of them may be finally selected as trim sections based on a user's input. According to another example, a part or a plurality of parts of a corresponding image may be selected by a user regardless of the provided trim section, and the selected part or a plurality of parts may be finally selected as the trim section.

When information about the highlight image is output as a result of image analysis, a highlight image may be generated by connecting some or all of one or more highlight sections according to a predetermined criterion. The generated highlight image may be output by being displayed on a display so that the user can identify it. Alternatively, information on the generated highlight image may be stored in a memory. The information on the highlight image is, for example, information capable of specifying each of the highlight sections constituting the highlight image, and may be information indicating the start and end of each highlight section, or the start and duration of each highlight section. It may also be information indicating (duration). In this case, the information may be expressed as the number of frames or time. Alternatively, the generated highlight image may be separately stored in the memory.

In order to configure the highlight image, some of the plurality of highlight sections may be selected according to a predetermined criterion. At this time, the predetermined criterion is, for example, a highlight section with the longest length, a section with a length closest to a predetermined length, a section with the smallest or the largest motion within the screen, a section with the smallest color diversity or the section with the greatest sharpness. A section or a section having a sharpness greater than or equal to a predetermined standard may be applied. In order to select a highlight section meeting each of the criteria, the above-described method may be similarly applied in the selection of the trim section.

Various criteria for selecting a highlight section constituting the highlight image may be applied alone or in combination with other criteria. When two or more criteria are combined and applied, different weights are assigned to each criterion, so that the priorities between the criteria can be different.

When there is a limitation on the total length of the highlight image or the number of highlight sections constituting the highlight image, only some of the highlight sections meeting the predetermined criterion may be selected according to priority by reflecting the same.

7 is a diagram for explaining a method of selecting a trim section from an image.

As shown in (a) of FIG. 7, after selecting the trim function of video editing for the selected image, the trim section can be manually selected by selecting the start time and the end time of the trim section according to a user input. . Finally, a trim image may be generated by a user input for determining the selected trim section. In this case, the user input may include a mouse input or a touch input. Alternatively, the user input may be a method of directly inputting a start time and end time or a start time and duration.

Alternatively, as shown in (b) of FIG. 7, the user can set the image analysis method according to the present invention to be executed by manipulating the setup menu of the application running on the user device (in the setting, the smart video trimmer is turned ON. Set). When the image analysis method according to the present invention is set to be executed, the image analysis method according to the present invention is performed under the above-described predetermined conditions, information on the trim section is generated, and stored in a memory as a result of analysis of the corresponding image. have.

Thereafter, when the analyzed image is selected by the user, an editing UI element, specifically, a trim button (eg, a magic wand button displayed at the lower left of the display screen of FIG. 7B) may be provided to the user. When the user selects the trim button, the automatically analyzed trim section may be immediately provided to the user based on information on the trim section stored in the memory. In this case, a portion corresponding to the automatically analyzed trim section may be displayed to be distinguished from a portion that does not correspond to the trim section by using a different frame brightness or a filter of a different color. Finally, a trim image may be generated by a user input for determining the provided trim section.

8 is a diagram for explaining an embodiment of providing a highlight image generated by performing the image analysis method of the present invention.

As described above, when the image analysis method of the present invention is performed, information on the highlight image may be output as a result of the image analysis. The information on the highlight image may include information for identifying a start time point and an end time point of a highlight section constituting the highlight image. Therefore, by using the information on the highlight image, it is possible to easily identify the highlight section and the non-highlight section in the analysis target image.

When a user selects an image for which image analysis has been completed according to the present invention, a highlight image composed of only the highlight section of the image may be provided to the user. As a modified example, as shown in FIG. 8, The selected image may be provided to the user by reproducing the highlight section at a normal speed, and reproducing the non-highlight section at a speed greater than or equal to the normal speed (eg, double speed).

Alternatively, as a result of performing image analysis according to the present invention, a key frame constituting a moving picture may be stored in a memory as a still image or provided to a user. For example, when photographing a moving object (dog, cat, etc.), it is not easy to obtain a still image with little shaking. In this case, a good still image of the subject can be secured by capturing a frame with high sharpness among them as a still image while reproducing a video in which the subject is captured.

According to the image analysis method of the present invention, since all frames constituting a video are analyzed, a key frame can be automatically obtained by selecting a frame with the highest sharpness among them. Specifically, similar to the request for a trim section proposal, when there is a request for a key frame proposal by the user, at least one of the frames included in the video may be determined as a key frame according to the image analysis method of the present invention. In this case, the key frame can be suggested to the user by displaying the key frame by distinguishing it from the frame other than the key frame. Accordingly, it is possible to save time and labor for a user to directly search for a key frame from a video.

The video analysis method according to the present invention includes, for example, when transmitting a video through an instant messenger application, when setting a video profile through SNS, when storing a video in a remote storage, and instant content sharing (e.g., LINE Timeline post) , LINE Timeline Story, etc.). In this case, when there is a limit on the length of the uploaded video, a section meeting the length limit may be selected and provided. The video length limit may be set differently for each application or service, such as 6 seconds for a video profile and 15 seconds for a story.

The highlight image provided as an analysis result of the image analysis method according to the present invention may be generated based on two or more images to be analyzed. For example, a highlight section may be identified by analyzing two or more analysis target images captured in the same travel destination, and a highlight image may be generated by connecting the identified highlight sections.

According to the present invention, a trim image or a highlight image may be automatically analyzed and generated from an image and provided to a user. Since the user can use the automatically generated trim video or highlight video, it is possible to save time and effort required for video editing.

In addition, since the image analysis according to the present invention is performed in consideration of the state of the user device and the usage pattern, it is possible to perform an image analysis task that requires a lot of resources at an optimal timing, thereby efficiently saving the resources of the user device. Can be used as.

Although the exemplary methods of the present disclosure are expressed as a series of operations for clarity of description, this is not intended to limit the order in which steps are performed, and each step may be performed simultaneously or in a different order if necessary. In order to implement the method according to the present disclosure, the exemplary steps may include additional steps, other steps may be included excluding some steps, or may include additional other steps excluding some steps.

Various embodiments of the present disclosure are not intended to list all possible combinations, but to describe representative aspects of the present disclosure, and matters described in the various embodiments may be applied independently or may be applied in combination of two or more.

In addition, the method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed by various computer devices and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known to and usable by a person skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, or the like.

The scope of the present disclosure is software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause an operation according to the method of various embodiments to be executed on a device or a computer device, and such software. Or it includes a non-transitory computer-readable medium (non-transitory computer-readable medium) that is stored in the instruction and the like that can be executed on a device or a computer.

Claims (20)

An image analysis method performed by a user device including a memory and at least one processor,
Determining whether to perform analysis of one or more images stored in the memory;
Selecting an image to be analyzed from among the one or more images;
Analyzing the image to be analyzed; And
And outputting an analysis result for at least one of the images to be analyzed. The method of claim 1,
Determining whether to perform the analysis of the image,
An image analysis method performed based on whether the user device is in an idle state. The method of claim 1,
Determining whether to perform the analysis of the image,
An image analysis method performed based on a usage pattern of the user device. The method of claim 1,
Determining whether to perform the analysis of the image,
When the user device is in an idle state and it is determined that the user device will not be used for a predetermined time based on a usage pattern of the user device, determining to analyze the image . The method of claim 1,
Selecting the analysis target image,
And selecting an image having the latest generation date among one or more images stored in the memory as the analysis target image. The method of claim 1,
Analyzing the analysis target image,
Analyzing each of a plurality of frames constituting the analysis target image;
Classifying at least one image section identified based on the analysis result of the frame into a highlight section and a non-highlight section; And
And analyzing each of the image sections classified as the highlight section. The method of claim 6,
Analyzing the frame,
The image analysis comprising the step of identifying a frame not to be output by determining whether at least one of a brightness value and a color value of pixels constituting each of the plurality of frames, and an amount of an edge in the frame meet a predetermined criterion Way. The method of claim 6,
The step of classifying the video section,
If the length of the video section is greater than or equal to a predetermined length, classifying the video section as a highlight section; And,
If the length of the video section is less than a predetermined length, the video analysis method comprising the step of classifying the video section as a non-highlight section. The method of claim 6,
Analyzing the image section,
An image analysis method comprising the step of calculating at least one of a result of analyzing each of a plurality of frames constituting an image section, a difference value between frames, and a length of a corresponding video section as a characteristic of the video section. The method of claim 9,
The step of outputting the analysis result,
And outputting information on at least one highlight section based on the characteristics of the video section. The method of claim 1,
The step of outputting the analysis result,
And outputting an analysis result for the corresponding image when an edit screen request for at least one of the analysis target images is received from a user of the user device. The method of claim 1,
The step of outputting the analysis result,
Receiving a selection of one or more images stored in the memory from a user of the user device;
When an edit screen request is received for the selected image, determining whether the selected image corresponds to the analysis target image and an analysis result of the selected image is stored in the memory; And
When it is determined that the analysis result for the selected image is stored, outputting the analysis result
Containing, image analysis method. The method of claim 12,
When it is determined that the analysis result for the selected image is stored, displaying an editing UI element for the selected image together with the analysis result. A computer program stored in a computer-readable recording medium for executing the image analysis method according to any one of claims 1 to 13 on a computer. A user device that performs an image analysis method, comprising:
A memory for storing one or more images; And
Including at least one processor,
The processor,
Determining whether to perform analysis of one or more images stored in the memory;
Selecting an image to be analyzed from among the one or more images;
Analyzing the image to be analyzed; And
A user device performing the step of outputting the analysis result. The method of claim 15,
The processor,
When the user device is in an idle state and it is determined that the user device will not be used for a predetermined period of time based on a usage pattern of the user device, determining to analyze the image. The method of claim 15,
The processor, in the step of analyzing the analysis target image,
Analyzing each of a plurality of frames constituting the analysis target image;
Classifying at least one image section identified based on the analysis result of the frame into a highlight section and a non-highlight section; And
The user device of performing the step of analyzing each of the image sections classified as the highlight section. The method of claim 17,
The processor, in the step of analyzing the image section,
Performing a step of calculating at least one of a result of analyzing each of the plurality of frames constituting the video section, a difference value between frames, and a length of the video section as a characteristic of the video section,
In the step of outputting the analysis result,
A user device for performing the step of outputting information on at least one highlight section based on the characteristics of the video section. The method of claim 15,
The processor, in the step of outputting the analysis result,
When an edit screen request for at least one of the analysis target images is received from a user of the user device, outputting an analysis result for the corresponding image is performed. The method of claim 15,
The processor, in the step of outputting the analysis result,
Receiving a selection of one or more images stored in the memory from a user of the user device;
When an edit screen request is received for the selected image, determining whether the selected image corresponds to the analysis target image and an analysis result of the selected image is stored in the memory; And
If it is determined that the analysis result for the selected image is stored, performing the step of outputting the analysis result.

Images