WO2024076202A1

WO2024076202A1 - Electronic device for generating reactive image on basis of comparison between plurality of frames, and method therefor

Info

Publication number: WO2024076202A1
Application number: PCT/KR2023/015417
Authority: WO
Inventors: 이철우
Original assignee: 이철우
Priority date: 2022-10-07
Filing date: 2023-10-06
Publication date: 2024-04-11

Abstract

An electronic device for generating a reactive image on the basis of a comparison between a plurality of frames, and a method therefor are disclosed. The electronic device according to the present disclosure comprises: a memory including an image; and a processor electrically connected to the memory, wherein the processor identifies the movement of at least one object on the basis of a comparison between a plurality of continuous frames included in the image, selects, as at least one target object, the at least one object of which the size of the movement is a threshold value or higher, identifies an input operation corresponding to the respective at least one target object, and connects the respective at least one target object to the identified input operation so as to generate a reactive image.

Description

Electronic device and method for generating a responsive image based on comparison between multiple frames

This disclosure relates to an electronic device and method for generating a responsive image. More specifically, the present disclosure relates to an electronic device and method for generating a responsive image based on comparison between a plurality of frames.

Recently, video recording technology has developed greatly. Not only camcorders and digital cameras, but also mobile devices such as smartphones can capture high-resolution images. Additionally, 360-degree cameras and 3D video cameras are appearing.

Videos are captured by a video recording device, stored in a specific format, and played back by a terminal capable of playing them. Video playback is provided unilaterally in chronological order without interaction with the viewer. In other words, the viewer can only feel the visual sense through the video being played.

The present disclosure provides a user interface for generating a responsive image by identifying the movement of at least one object based on comparison between a plurality of frames included in the image and connecting the at least one object with a corresponding input manipulation. We would like to provide a device.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

An electronic device according to the present disclosure for achieving the above-described technical problem includes a memory including an image, and a processor electrically connected to the memory, wherein the processor performs comparison between a plurality of consecutive frames included in the image. Based on this, identify the movement of at least one object, select the at least one object whose movement size is greater than or equal to a threshold value as at least one object, identify an input operation corresponding to each of the at least one object, and A responsive image can be generated by connecting each of at least one object to the identified input manipulation.

In addition, the responsive video method based on comparison between a plurality of frames according to the present disclosure to achieve the above-described technical problem identifies the movement of at least one object based on comparison between a plurality of consecutive frames included in the video. selecting the at least one object whose motion magnitude is greater than or equal to a threshold as at least one object, identifying an input operation corresponding to each of the at least one object, and each of the at least one object. It may include generating a responsive image in connection with the identified input manipulation.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the means for solving the above-described problem of the present disclosure, by analyzing the movement of at least one object appearing in the image and providing a user interface for creating a responsive image, user accessibility to the responsive image is improved and intuitive Can provide user experience.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is a block diagram showing the configuration of an electronic device according to the present disclosure.

FIG. 2 is a flowchart illustrating the operation of an electronic device that generates a responsive image based on comparison between a plurality of frames according to the present disclosure.

Figure 3 illustrates the process of generating a responsive image based on feature points according to the present disclosure.

Figure 4 illustrates a user interface for creating a responsive image according to the present disclosure.

Figure 5 illustrates the process of generating a responsive video using a machine learning model according to the present disclosure.

Like reference numerals refer to like elements throughout this disclosure. The present disclosure does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present disclosure pertains is omitted. The term 'unit, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'unit, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Singular expressions include plural expressions unless the context clearly makes an exception. The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Before explaining the operating principles and embodiments of the present disclosure with reference to the attached drawings, some terms will be explained as follows.

Content (content or content) may include various contents provided visually, such as videos, still images, holograms, etc., and may include various contents provided through auditory, gustatory, olfactory, etc. The example is not limited to this. Additionally, content may be provided in virtual reality (VR), but the embodiment is not limited thereto.

Objects included in content may be included in part of the content or may include the entire content. For example, if the content is a video, it includes various objects located within the entire frame or part of the frame of the video. It can be done, and it can also mean the video itself.

Responsive content is a command that triggers a 'reaction (reaction, feedback, etc.)' of an object included in the content (which can be called an input operation, such as a reserved command including touch operation, sound command, motion command, etc.) ) is input, it may include various contents related to the corresponding object. Hereinafter, the description will be made on the assumption that the responsive content according to the present disclosure is a responsive video, and the type of responsive content according to the present disclosure is not limited to being a video.

Here, the response may include movement of the object, change in shape/shape of the object, occurrence of a specific event, and/or occurrence of change in content according to a command (input operation), but the embodiment is not limited thereto.

Responsive video refers to a video that is played in a form corresponding to a command (eg, touch operation) by a user (ie, viewer) watching the video. For example, a responsive image may refer to an image in which the movement of touching an object is played when a user operation in the form of touching a specific object (for example, a pillow) is applied to the touch screen. Also, for example, responsive video refers to an image in which when a user manipulation in the form of pressing a specific object is applied to the touch screen, the movement of the object being pressed and the movement restored after the user manipulation are played. You can.

A 'command for triggering a response' (input manipulation) of an object included in content may include a user manipulation of the content received through an input means of a computer that provides a responsive image. For example, the user operation is an operation that can be input at a specific point or area in the content through an input means such as a mouse or a touch screen (e.g., a click operation, a drag operation, a contact touch operation for a certain period of time, or a force touch) It may include manipulation (i.e., touch manipulation of applying specific pressure to a touch screen or touch pad), etc.). In addition, for example, the user operation involves the arrangement or movement of the terminal itself, which can be obtained by using a sensor (e.g., acceleration sensor, gyro sensor, etc.) provided by the computer (or terminal) as an input means. It can be included. When the content is VR content, a command for triggering a reaction may be performed by sensing the movement of a worn terminal or manipulating a terminal such as a joystick, but the embodiment is not limited thereto.

Below, a method for creating a responsive image will be described. The creation of a responsive image will be described assuming that it is performed by a processor of a device (eg, a computer).

The processor 1410 may determine a command to trigger a response of an object included in one or more original images. Here, the original video can be called a basic video, and the original video may be content that is not implemented in a responsive manner. For example, the original video may be captured content and may include a combination of a plurality of frames storing frames for each position of an object in space. The original video may be content collected through communication, may be three-dimensional content, or may be VR content, but the embodiment is not limited thereto.

The processor 1410 may receive an input specifying a frame section of the original image, and the frame section may include a specific section to be implemented in a responsive manner among all frames of the original image. The frame section can be set by the user through various methods. In an embodiment, the processor 1410 processes the starting frame of the original video from the user (i.e., the first frame in the time domain to be produced as a responsive image) to the final frame (i.e., the last frame in the time domain to be produced as a responsive image). You can be chosen. Additionally, in an embodiment, a time section may be designated by the user.

The processor 1410 is connected to the responsive image and can directly receive a command from the user to trigger a reaction of the object. For example, in the case of a device equipped with a touch screen, the processor 1410 provides a process for receiving a specific input operation from the user, and may receive a specific command according to an object moving on the touch screen during the process. .

Additionally, the processor 1410 may receive a user's selection of a command type to be linked to the responsive image and receive an operation that can replace the corresponding command type. For example, when creating a responsive video on a device with a touch screen and using a computer (including VR devices) that does not have a touch screen, the computer uses mouse operations instead of touch operations on the touch screen. By receiving it, you can create a responsive video.

The command for triggering the object's response may match or correspond to the movement of the object included in the frame section. The location or area where the command for triggering the reaction of the object is set to be input may correspond to the area corresponding to the movement of the object within the frame included in the frame section.

The processor 1410 may apply a method of creating a virtual layer in the entire area or a specific area of each frame within a designated frame section of the original image in order to connect an object and a command for triggering the object's response. A virtual layer may refer to a layer that is overlaid on the frames that make up the original image and that can receive user input without being visually expressed on the screen.

The processor 1410 is a command to trigger a response of an object by moving a specific area on the frame (i.e., the path on which the object moves within the frame section) (e.g., moving the mouse cursor through mouse operation or using the first touch screen on the touch screen). In the case of a drag operation from a point to a second point, a virtual layer composed of a specific number of detailed cells can be created on the frame.

Additionally, the processor 1410 may generate a virtual layer composed of a plurality of detailed cells corresponding to the frame section. The processor 1410 can calculate the number of frames included in the frame section, apply the number of frames in the frame section as the number of detailed cells, and sequentially match each frame in the frame section to each detailed cell. .

For example, when the processor 1410 wants to generate content so that n frames are variably played (i.e., manipulated) according to a command to trigger a response of an object, the processor 1410 divides a specific area into n detailed cells. It can be divided.

Thereafter, the processor 1410 may match each frame to each divided detailed cell so that the matched frame is provided when a specific detailed cell is selected or designated. That is, when an object (e.g., a hand) moves in a specific direction and a virtual layer is created along the movement path of the object, the processor 1410 operates each frame section in order, starting from the detail cell at the first point where the object begins to move. It can be matched to the frame of .

Additionally, the processor 1410 may generate a plurality of detailed cells constituting the virtual layer with different sizes or spacing. If the speed at which an object moves changes during a frame section in the original video, if the virtual layer is divided into detail cells of the same size, the position of the object in the frame and the position of the detail cells may not match. Accordingly, the processor 1410 may vary the size or spacing of detailed cells to match the movement of the object. In other words, when playing video content, the speed changes at the same time interval to obtain frames of moving objects, so in fast-moving sections, the gap between object positions within successive frames is large, and in slow movements, the gap between object positions within successive frames is narrow. do. Therefore, the processor 1410 must generate a plurality of detailed cells to match the object spacing within the frame so that the position of the input operation (command for triggering the object's response) entered by the user matches the position of the object within the frame. .

Additionally, the processor 1410 may determine the length of the virtual layer. In an embodiment, the processor 1410 can determine the location of each object (or a specific feature point of an object) within a frame section and recognize the path on which the object moves, and the processor 1410 can create a virtual path with a length including the path. Layers can be formed. Additionally, the processor 1410 may determine the shape of the virtual layer and detailed cell.

Responsive video may be content that has been filmed and stored in advance, or it may be content that adds or synthesizes additional content to the original video. For example, a responsive video may include a video, still image, hologram, etc., and additional content may be played together when the original video is played, turning the original video into a responsive video.

Additionally, a responsive video may include multiple responsive videos, which may mean that the content changes or plays in a form corresponding to a specific input operation by a user (i.e., viewer) watching the content. You can. For example, it may mean an image that is played back as if the user's input operation corresponding to a specific movement of the captured original image is connected to the object in the image moving according to the user's operation.

In an embodiment, a compressed image refers to an image compressed into a minimum movement unit to implement a basic image as a responsive image. For example, if the basic image contains the same movement repeatedly, the compressed image deletes the repeated movement and leaves only one. Additionally, for example, if the base image includes both movement from the first position to the second position and movement from the second position to the first position, the compressed image is the base image moving from the first position to the second position. Leaving it alone, movement from the second position to the first position can play the remaining basic image in the reverse direction.

In addition, a multi-responsive video creation file is a content file created by compressing a plurality of basic images and can play various actions according to the user's manipulation, or can be implemented as a multi-responsive video by being played together with one or more basic images. This means metadata that can be used. In an embodiment, the processor 1410 may generate a responsive image without generating a compressed image even if duplication is allowed.

In an embodiment, the processor 1410 generates or outputs a responsive image from a base image (the base image may be a responsive image) by not using compressed images and using all repeated movements as is without deleting them. You can.

Hereinafter, a method for generating a multi-responsive video according to the present disclosure will be described, assuming that the content is an video.

The processor 1410 may acquire a basic image. The basic image may be an original image that includes the movement of an object to be implemented in a responsive manner according to the user's manipulation. Responsive video creators (e.g., content providers or individual users) can shoot videos containing actions they want to implement in a responsive manner.

In an embodiment, the processor 1410 may obtain images of multiple movements of the same object from the user and then generate them in the form of multiple responsive images. For example, when the object is the user's hand, the control module 1500 (190) can acquire a plurality of images of the user's index finger moving or bending in various directions while spread.

Additionally, if you want to create a responsive video that rotates your head up, down, left, right, or changes facial expressions in response to a specific manipulation, you can capture an image that includes all of the desired head movements and expressions.

In addition, if you want to create a responsive video in which a water balloon bursts or bounces up from the floor according to a manipulation input by the user, the user must drop a water balloon of the same color and size so that the first image of the popping water balloon and the water balloon are You can sequentially film a second video that bounces without exploding.

Additionally, after performing a specific movement, one basic image in which a different event occurs rather than a repetition of an existing movement can be obtained. That is, the processor 1410 may acquire an image in which a plurality of events occur with respect to an object as a base image.

The processor 1410 may generate a compressed image based on the basic image. The compressed video may be responsive and include only the movement of the object to be implemented according to the user's manipulation. In embodiments, images that allow duplication may also be applied instead of compressed images.

The processor 1410 can receive multiple responsive image creation conditions for compressed images. The multi-responsive image creation condition may be a plurality of manipulation inputs corresponding to responses that can be generated from the compressed image.

The processor 1410 may generate a stack structure of the compressed image, where each extraction area (e.g., a first extraction area and a second extraction area) may include a plurality of stacks for different events. For example, a first stack represented by a solid line and a second stack represented by a dotted line may be included in each extraction area. The processor 1410 may be implemented by determining the stack to be executed among the first stack and the second stack based on the location where the first operation is input to each extraction area from the user.

Additionally, the first event and the second event in each extraction area may include overlapping pixels, and the processor 1410 may leave only one of the overlapping pixels among the stack for the first event and the stack for the second event. . Even if the compressed image contains only one data about an overlapping pixel within a specific extraction area, one of the first event and the second event depending on the user's next operation (e.g., change in the direction of movement of the touch operation or the intensity of applied pressure) The stack can be determined. Through this, the computer can create compressed images with minimal data.

Below, a user interface for creating a responsive image according to an embodiment of the present disclosure will be described. Responsive images may include the above-described multi-type responsive images and multi-dimensional responsive images.

1 is a block diagram showing the configuration of an electronic device according to the present disclosure. The electronic device may be understood as an example of a computer in this specification.

In one embodiment, the electronic device 1400 may include a processor 1410, a memory 1420, a user input unit 1430, and/or a display unit 1440. The components shown in FIG. 14 are not essential for implementing the electronic device 1400 according to the present disclosure, so the electronic device 1400 described herein may have more or fewer components than the components listed above. It can have elements.

In one embodiment, the processor 1410 includes a memory that stores data for an algorithm for controlling the operation of components within the device or a program that reproduces the algorithm, and performs the above-described operations using the data stored in the memory. It can be implemented with at least one processor. At this time, the memory and processor may each be implemented as separate chips. Alternatively, the memory and processor may be implemented as a single chip.

In addition, the processor 1410 may control any one or a combination of the components described above in order to implement various embodiments according to the present disclosure described below on the present device.

In one embodiment, the memory 1420 may store data supporting various functions of the device and a program for the operation of the processor 1410, and may store input/output data (e.g., music files, Images, videos, etc.) can be stored, and a number of application programs (application programs or applications) running on the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The memory 1420 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), or a multimedia card micro type. micro type), card type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable) It may include at least one type of storage medium among programmable read-only memory (PROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. Additionally, the memory 1420 is separate from the device, but may be a database connected wired or wirelessly.

In one embodiment, the memory 1420 may include an image for generating a responsive image. For example, an image may be composed of a plurality of consecutive frames.

In one embodiment, memory 1420 may include machine learning model 1425. The machine learning model 1425 can use a deep learning method based on a deep neural network. For example, the machine learning model 1425 may be based on a convolution neural network (CNN) method.

In one embodiment, the user input unit 1430 is used to receive information from the user. When information is input through the user input unit, the processor 1410 can control the operation of the device to correspond to the input information. The user input unit 1430 includes hardware-type physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc. located on at least one of the front, back, and sides of the device) and software-type keys. May include touch keys. As an example, the touch key consists of a virtual key, soft key, or visual key displayed on a touch screen-type display unit through software processing, or is displayed on the touch screen. It may be composed of touch keys placed in other parts. Meanwhile, the virtual key or visual key can be displayed on the touch screen in various forms, for example, graphic, text, icon, video or these. It can be made up of a combination of .

In one embodiment, the display unit 1440 may implement a touch screen by forming a mutual layer structure or being integrated with the touch sensor. This touch screen functions as a user input unit that provides an input interface between the device and the user, and can simultaneously provide an output interface between the device and the user. That is, the user input unit 1430 and the display unit 1440 can be integrated into each other and implemented as a touch screen.

The display unit 1440 displays (outputs) information processed by the device. For example, the display unit display unit 1440 displays execution screen information of an application program (for example, an application) running on the device, or UI (User Interface) and GUI (Graphic User Interface) information according to this execution screen information. can be displayed.

In one embodiment, the display unit 1440 may be used as an input means. For example, the user may perform an operation that can be input at a specific point or area in the image through the display unit 1440 (e.g., a click operation, a drag operation, a contact touch operation over a certain period of time, a force touch operation (i.e., a touch operation) You can perform input operations such as touch operations (applying specific pressure to the screen or touch pad, etc.).

In one embodiment, the processor 1410 may generate a responsive image based on the image stored in the memory 1420. The processor 1410 may identify the movement of at least one object based on comparison between a plurality of frames included in the image. A method for identifying the movement of at least one object will be described later.

In one embodiment, the processor 1410 may generate a responsive image by connecting the movement of at least one identified object with a corresponding input manipulation. Specifically, the processor 1410 may generate at least one playback type by connecting each movement of at least one object with an input manipulation. At least one playback type may constitute a responsive video (e.g., multi-responsive video). For example, when the electronic device 1400 receives an input manipulation for an object included in one screen of a responsive video from a user, the processor 1410 may play a playback type corresponding to the input manipulation. In this case, the input manipulation can be understood as an input manipulation linked to the movement of the identified object.

In one embodiment, the processor 1410 may provide a user interface for creating a responsive image. For example, the processor 1410 may display a user interface through the display unit 1440. The user interface may include at least one playback type generated from the video. The processor 1410 may generate a responsive image based on the playback type selected by the user.

At least one component may be added or deleted in response to the performance of the components shown in FIG. 1. Additionally, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the system.

Meanwhile, each component shown in FIG. 1 refers to software and/or hardware components such as Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).

FIG. 2 is a flowchart illustrating the operation of an electronic device that generates a responsive image based on comparison between a plurality of frames according to an embodiment of the present disclosure. In FIG. 2 , the operation of the electronic device 1400 may be understood as being substantially performed by the processor 1410.

In operation 1510, the processor 1410 of the electronic device 1400 may identify the movement of at least one object based on comparison between a plurality of consecutive frames included in the image.

In one embodiment, the processor 1410 may identify at least one object included in the first frame of the image. The processor 1410 may extract feature points of the first frame or use a machine learning model 1425 to identify at least one object. The feature point of the first frame may be understood as relating to at least one object.

In one embodiment, the processor 1410 may identify the movement of at least one object by comparing the first frame with a second frame following the first frame. In this way, the processor 1410 can identify the movement of at least one object through comparison between successive frames for all of the plurality of frames included in the image. The processor 1410 may use feature points extracted from each frame or use a machine learning model 1425 to identify the movement of at least one object. The specific operation of the processor 1410 in operation 1510 will be described later.

In operation 1520, the processor 1410 may select at least one object whose motion size is greater than or equal to a threshold value as at least one object. In one embodiment, the processor 1410 may calculate the total distance an object moves on the screen as the size of the movement. For example, the processor 1410 may calculate the magnitude of movement based on Equation 1 below.

In one embodiment, processor 1410 may compare the magnitude of movement for each object to a threshold. The processor 1410 may ignore objects whose motion magnitude is less than a threshold value. The processor 1410 may select an object whose movement size is greater than or equal to a threshold value as the target object.

In operation 1530, the processor 1410 may identify an input manipulation corresponding to each of at least one object. In one embodiment, the processor 1410 may determine the type of movement of each of at least one object and identify an input manipulation corresponding to the type of movement. For example, when the object moves from bottom to top, the processor 1410 may correspond to the movement of the object to a swiping input from bottom to top. For example, when an object moves from a distance to a near distance and increases in size within the image, the processor 1410 may correspond to the movement of the object to a pinch out input.

In operation 1540, the processor 1410 connects each of the at least one object to the identified input manipulation to generate a responsive image. More specifically, the processor 1410 may generate a playback type consisting of at least one object and an input manipulation associated with the movement of the at least one object. The processor 1410 may generate a responsive image based on the generated playback type. In one embodiment, the processor 1410 may provide at least one playback type through a user interface for generating a responsive image and generate a responsive image based on the playback type selected by the user.

In one embodiment, the processor 1410 may display the generated responsive image through the display unit 1440. The processor 1410 may receive an input manipulation for an object of a responsive image and play a designated playback type. For example, when receiving a pinch-out input for an object, the processor 1410 may play a playback type that increases the size of the object.

Figure 3 illustrates a process for generating a responsive image based on feature points, according to an embodiment of the present disclosure.

In one embodiment, the image 1600 may include multiple frames. At least one

object

1602, 1604, and 1606 may appear in the image 1600.

Reference number 1610 indicates a comparison operation for some

frames

1612 and 1614 among a plurality of frames. The N+1-th frame 1614 can be understood as a frame that follows the N-th frame 1612. Although two frames are shown at reference number 1610, the operation of the processor 1410 at reference number 1610 may be understood as being performed on all frames included in the plurality of frames.

In one embodiment, the processor 1410 may extract feature points of the first frame 1612 and the second frame 1614. A feature point may be understood as relating to at least one

object

1602, 1604, and 1606. The processor 1410 may identify the movement of at least one

object

1602, 1604, and 1606 based on the difference in feature points between the first frame 1612 and the second frame 1614. For example, the processor 1410 may identify that the object 1602 has moved to the upper left corner based on a change in the position of a feature point for the object 1602. For example, the processor 1410 may identify that the object 1604 has moved to the bottom left based on a change in the position of a feature point for the object 1604. For example, the processor 1410 may identify that the object 1606 has moved to the bottom left based on a change in the position of a feature point for the object 1606.

In one embodiment, processor 1410 may collect the magnitude of movement of at least one

object

1602 , 1604 , and 1606 identified at reference numeral 1610 in operation 1620 . For example, the magnitude of the movement may be calculated as the total distance that at least one

object

1602, 1604, and 1606 moves on the screen.

In one embodiment, the processor 1410 may compare the movement of at least one

object

1602, 1604, and 1606 to a threshold value in operation 1630. The processor 1410 may identify an object whose movement is greater than the threshold value. For example, the motion of

objects

1602 and 1606 may be greater than the threshold, and the motion of object 1604 may be less than the threshold. The processor 1410 may select the

objects

1602 and 1606 as at least one object.

In one embodiment, the processor 1410 may identify an input manipulation that corresponds to a type of movement of at least one object. The processor 1410 may generate at least one

playback type

1602 or 1604 by connecting each movement of at least one object to an input manipulation. For example, playback type 1640 may correspond to movement of object 1602. The input operation in playback type 1640 may be a swipe operation from right to left. For example, playback type 1645 may correspond to movement of object 1606. The input operation in the playback type 1645 may be a swiping operation to the bottom left.

In one embodiment, the processor 1410 may provide a user interface for generating a responsive image (operation 1650). The processor 1410 may provide

playback types

1640 and 1645 through a user interface and generate a responsive image 1660 based on the user's selection of the

playback types

1640 and 1645. A description of the user interface may be referred to as a description of FIG. 4 .

Figure 4 illustrates a user interface for creating a responsive image according to an embodiment of the present disclosure.

The processor 1410 may provide a user interface screen 1700 for generating a responsive image through the display unit 1440. The processor 1410 may automatically recognize images stored in the memory 1420, such as the screen 1710, in response to user input. Automatic recognition can be understood as a series of operations to create a responsive image, as shown in FIG. 2.

The processor 1410 may display at least one playback type identified through the image, as shown on the screen 1720. For example, the screen 1720 may indicate a playback type that links the movement of at least one object (e.g., the mouth of a glass bottle) to an input operation of swiping from bottom to top. The processor 1410 may receive a user's input on the screen 1720 and identify a playback type to generate a responsive image. The processor 1410 can display the playback type selected by the user, as shown in the screen 1730, and generate a responsive image based on it.

Figure 5 illustrates a process for generating a responsive image using a machine learning model according to an embodiment of the present disclosure.

In one embodiment, an image for generating a responsive image may include a plurality of

frames

1800, 1802, 1804, 1806, 1808, 1810, and 1812. Processor 1410 may use machine learning model 1425 to generate a playback type. The machine learning model 1425 can be learned using an image database including the movement of the object. For example, the machine learning model 1425 may be a deep learning model for identifying the movement of at least one object included in an image.

In one embodiment, the processor 1410 may identify the location of at least one

object

1602, 1604, and 1606 in the first frame 1800 using the machine learning model 1425. The machine learning model 1425 may identify the movement of at least one

object

1602, 1604, and 1606 in a plurality of consecutive frames. The machine learning model 1425 may identify an object whose movement size is higher than a threshold among the movements of at least one identified

object

1602, 1604, and 1606 as an object.

In one embodiment, the machine learning model 1425 may detect the movement of the object 1602 in the third frame 1804 to the sixth frame 1810 and identify the object 1602 as an object. Processor 1410 may generate a first playback type 1820 by combining movement of object 1602 with a swiping input to the upper left.

In one embodiment, the machine learning model 1425 may detect the movement of the object 1606 in the fourth frame 1806 to the fifth frame 1808 and identify the object 1606 as an object. Processor 1410 may generate a second playback type 1830 by combining movement of object 1606 with a swiping input to the bottom left.

In one embodiment, the processor 1410 may provide a first playback type 1820 and a second playback type 1830 through a user interface. The processor 1410 may generate a responsive image including at least one of the first playback type 1820 or the second playback type 1830 based on user selection.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, etc.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which this disclosure pertains will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

In electronic devices,

memory containing images; and

Including a processor electrically connected to the memory,

The processor,

Identifying the movement of at least one object based on comparison between a plurality of consecutive frames included in the image,

Selecting the at least one object whose motion size is greater than or equal to a threshold value as at least one object,

Identifying an input operation corresponding to each of the at least one object,

An electronic device that generates a responsive image by connecting each of the at least one object to the identified input manipulation.
According to claim 1,

The responsive image is a multi-responsive image, an electronic device.
According to clause 2,

The processor,

generate at least one playback type based on the at least one object and an input manipulation associated with each of the at least one object;

An electronic device that provides the at least one playback type through a user interface.
According to clause 3,

The processor,

Receiving user input through the user interface,

An electronic device that generates the responsive image based on the playback type selected by the user input.
According to clause 4,

The processor,

Extracting feature points of the plurality of frames,

An electronic device that identifies movement of the at least one object based on a change in position of the feature point.
According to clause 4,

The memory includes a machine learning model,

The processor,

An electronic device that identifies the at least one object in the plurality of frames using the machine learning model and detects movement of the at least one object between the plurality of frames.
According to clause 6,

The processor,

An electronic device that extracts at least one frame in which movement of the at least one object among the plurality of frames is detected as the playback type using the machine learning model.
According to clause 7,

further comprising a display,

The processor,

An electronic device that calculates the total distance that the at least one object moves on the display as the size of the movement.
In a method performed by an electronic device,

Identifying movement of at least one object based on comparison between a plurality of consecutive frames included in the image;

selecting the at least one object whose motion size is greater than or equal to a threshold value as at least one object;

identifying an input manipulation corresponding to each of the at least one object; and

A method of generating a responsive image based on comparison between a plurality of frames, including the step of connecting each of the at least one object to the identified input operation to generate a responsive image.
A computer-readable recording medium coupled to a computer and storing a program for executing the method of claim 9.