KR20220056769A - Electronic device and method for controlling thereof - Google Patents

Abstract

Disclosed are an electronic device and a control method thereof. The electronic device of the present disclosure includes: a first camera including a first lens having a first focal distance; a second camera including a second lens having a second focal distance different from the first focal distance; a memory; and a processor acquiring information about a first object included in a first image acquired by photographing the surroundings of an electronic device through the first camera, and, then, when identifying that information about a second object associated with the first object based on the acquired information about the first object, acquiring a second image through the second camera, and determining a time to perform high-speed photography through the first camera based on whether the second object is included in the acquired second image. Therefore, the present invention is capable of accurately and efficiently determining a time to perform an SSM function.

Description

ELECTRONIC DEVICE AND METHOD FOR CONTROLLING THEREOF

The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device capable of determining when to perform an SSM function based on an object included in an image, and a method for controlling the same.

An SSM (Super Slow Motion) image refers to an image captured to record high-speed motion in a specific scene. Since a large amount of computing resources, storage resources, and power are consumed to generate the SSM image, it is generally preferred to generate the SSM image within a short time period in which a specific event occurs within a scene.

On the other hand, when the SSM video recording function is manually activated, when an unexpected event occurs or an object included in the video moves quickly, there is a possibility that the user performs the SSM video recording start operation earlier or later than the appropriate timing. do. Accordingly, there is a possibility that the user may not be able to record an SSM image for an intended event or record an SSM image with a very large capacity because an unintended scene is included.

The present disclosure has been devised to solve the above-described problems, and an object of the present disclosure is to determine a time point at which an SSM function is performed based on an object included in an image obtained through at least one camera, and a method for controlling the same is to provide.

In one embodiment of the present disclosure, the electronic device includes a first camera including a first lens having a first focal length, a second camera including a second lens having a second focal length different from the first focal length; Obtaining information on a first object included in a first image obtained by photographing a periphery of the electronic device through a memory and the first camera, and obtaining the first object based on the obtained information on the first object If it is identified that there is information on the second object associated with The processor may include a processor that determines when to perform high-speed imaging using the camera.

The method of controlling an electronic device includes a first camera including a first lens having a first focal length and a second camera including a second lens having a second focal length different from the first focal length. obtaining information on a first object included in a first image obtained by photographing a periphery of the electronic device through a camera; a second associated with the first object based on the obtained information on the first object If it is identified that there is information on the object, using the first camera based on whether the second object is included in the step of obtaining a second image through the second camera and the obtained second image to determine a point in time to perform high-speed imaging.

According to various embodiments as described above, the electronic device may accurately and efficiently determine a time point at which the SSM function is performed.

1A is a block diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
1B is a diagram for explaining the structure of a knowledge base according to an embodiment of the present disclosure;
2 is a flowchart for explaining a method of controlling an electronic device according to an embodiment of the present disclosure;
3 is a flowchart illustrating a process in which an electronic device activates a second camera according to an embodiment of the present disclosure;
4A and 4B are flowcharts for explaining a process in which an electronic device performs high-speed imaging using a first camera, according to an embodiment of the present disclosure;
5 is a flowchart illustrating a process in which an electronic device performs high-speed image capturing using a first camera according to an embodiment of the present disclosure;
6 is a flowchart for explaining a process of performing high-speed imaging using a first camera, according to an embodiment of the present disclosure;
7 is a flowchart for explaining a process of performing high-speed imaging using a first camera, according to an embodiment of the present disclosure;
8 is a block diagram illustrating in detail the configuration of an electronic device according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the drawings. Various elements and regions in the drawings are schematically drawn. Accordingly, the technical spirit of the present invention is not limited by the relative size or spacing drawn in the accompanying drawings.

1A is a block diagram illustrating a configuration and operation of an electronic device 100 according to an embodiment of the present disclosure. In describing the present disclosure, the electronic device 100 may be implemented as a user terminal device such as a smart phone, a tablet PC (Personal Computer), a desktop PC, a laptop PC, a netbook computer, and a wearable device, but is limited thereto. it is not going to be The electronic device 100 of the present disclosure may be implemented as various devices including a camera.

1A , the electronic device 100 may include a first camera 110 , a second camera 120 , a memory 130 , a processor 140 , and a display 150 . However, the configuration shown in FIG. 1A is an exemplary diagram for implementing embodiments of the present disclosure, and appropriate hardware and software configurations at a level obvious to those skilled in the art may be additionally included in the electronic device 100 .

The first camera 110 is configured to acquire one or more images by photographing the surroundings of the electronic device 100 . The first camera 110 may acquire a first image by photographing the surroundings of the electronic device 100 . That is, the first camera 110 may acquire a first image composed of a plurality of image frames photographed around the electronic device 100 , and the first image may include a live view. can In addition, the first image composed of a plurality of image frames may be stored in a buffer of the memory 130 for temporarily storing information. The first camera 110 may include a first lens having a first focal length.

The second camera 120 is configured to acquire one or more images by photographing the surroundings of the electronic device 100 like the first camera 110 . The second camera 120 may acquire a second image including a plurality of image frames obtained by photographing the periphery of the electronic device 100 , and the second image may include a live view.

Meanwhile, the second camera 120 may be located within a threshold distance from the first camera 110 on the electronic device 100 . The critical distance may be a value preset through experiments or research, but is not limited thereto, and may be changed in the manufacturing stage of the electronic device 100 .

The second focal length of the second lens included in the second camera 120 may be different from the second focal length of the first lens included in the first camera 110 . For example, the second focal length of the second lens included in the second camera 120 may be shorter than the first focal length of the first lens included in the first camera 110 .

For example, the second camera 120 may include an ultra-wide lens, and the first camera 110 may include a wide-lens. That is, the second camera 120 may have a larger field of view (FOV) than the first camera 110 . Accordingly, when the first camera 110 and the second camera 120 photograph the periphery of the electronic device 100 at a critical distance from each other, the second image acquired through the second camera 120 is the first It may be composed of an image frame including a wider area than the first image acquired through the camera 110 .

However, this is only an example, and the second focal length of the second lens included in the second camera 120 may be longer than the first focal length of the first lens included in the first camera 110 . For example, the second camera 120 may include a wide-angle lens, and the first camera 110 may include an ultra-wide-lens.

Each of the first camera 110 and the second camera 120 may include an image sensor (eg, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor, etc.). The image sensor may be turned off or on depending on whether power is applied or the like.

The state in which each camera 110 and 120 is deactivated is a state in which the image sensor included in each camera 110 and 120 is turned off or the image sensor included in each camera 110 and 12 is turned on, but each camera (110, 120) may mean a state in which a plurality of image frames acquired using the image sensor are not stored in the buffer.

When the respective cameras 110 and 120 are activated, the image sensors included in the respective cameras 110 and 120 are turned on, and a plurality of image frames obtained by the respective cameras 110 and 120 using the image sensors are buffered. It may mean a state in which a plurality of image frames stored in the buffer are being displayed on the display 150 by the processor 140 . In describing the present disclosure, the activation of the second camera 120 means that the second camera 120 is switched from a deactivated state to an activated state.

The memory 130 may store data related to at least one other component of the electronic device 100 or at least one instruction. The command means one action statement that can be directly executed by the processor 140 in a programming language, and is a minimum unit for program execution or operation. The memory 130 is accessed by the processor 140 , and reading/writing/modification/deletion/update of data by the processor 140 may be performed.

In the present disclosure, the term "memory" refers to a memory 130, a ROM (not shown) in the processor 140, a RAM (not shown), or a memory card (not shown) mounted in the electronic device 100 (eg, micro SD). card, memory stick).

The memory 130 includes a software module for determining when to perform a Super Slow Motion (SSM) function through the first camera 110 based on the first image and the second image, and a software module for each module to perform various operations. You can store the data you need. The module for determining when the first camera 110 performs the SSM function includes the first region of interest determination module 20 , the object recognition module 30 , the object property analysis module 40 , and the related object determination module 50 . ), a second image monitoring module 60 and a high-speed imaging performing module 70 may be included.

The SSM function captures the surroundings of the electronic device 100 at high-speed (eg, 480 fps (frames per second) or 960 fps) to obtain a super slow motion image, and obtains the obtained super slow motion image. It refers to a function of slow motion video playback (eg, 32 times slower than normal video playback speed (30 fps) or 4 times slower than normal slow motion (240 fps)). The SSM function allows users to efficiently detect rapid movements or minute changes in objects that are difficult to identify with the naked eye. Meanwhile, the playback speed or shooting speed related to the SSM function is not limited to the above-described exemplary values and may be set by the user to various values.

The super slow motion image captured by performing the SSM function through the first camera 110 may be an image having a relatively high frame rate (eg, 480 fps or 960 fps). Meanwhile, the frame rate corresponding to the SSM function may be a preset value, but may be changed by a user.

The first region of interest (ROI) determination module 20 is configured to determine a first region of interest on the first image acquired through the first camera 110 and obtain information on the determined first region of interest. It is a module. Here, the first region of interest of the first image refers to a partial region determined (or selected) for image analysis, etc. in the first image.

The first ROI-determining module 20 may automatically determine the first ROI on the first image using the third model 95 learned to determine the ROI or an auto focus method. As another example, the first ROI-determining module 20 may determine a region to which a user touch is input on the first image displayed on the display 150 as the first ROI.

The first ROI-determining module 20 may acquire information on the determined first ROI. For example, the information on the first region of interest includes feature information of an image included in the first region of interest (eg, pixel values, luminance and chromaticity information, etc. of an image included in the first region of interest) and the first region of interest. Information on the size or shape of an indicator (eg, a bounding box corresponding to the first ROI) may be included. The indicator indicating the first ROI may have a preset size or shape, but is not limited thereto and may be changed by a user command. In addition, the first ROI-determining module 20 may acquire characteristic information of the image included in the first ROI by analyzing the image included in the first ROI.

The object recognition module 30 is a module for recognizing a first object included in the first ROI by using the first model 80 to obtain information about the first object. Specifically, the object recognition module 30 inputs information on the first region of interest obtained through the first region of interest determining module 20 into the first model 80, and the first object included in the first region of interest information can be obtained. The information on the first object may include, for example, the type of the first object, the size of the first object, and information on the amount of change in the position of the first object based on a plurality of frame images included in the first image. That is, the first object recognition module 30 may recognize the first object included in the first ROI by acquiring information about the first object.

The object property analysis module 40 analyzes the properties (dynamic or static properties) of the first object based on the information about the first object obtained through the object recognition module 30 to analyze the first object This module identifies whether or not is a moving object. For example, the object property analysis module 40 identifies the amount of change in the position of the first object on the plurality of image frames constituting the first image among the information on the first object, and based on the amount of change in the position of the identified first object Thus, the properties of the first object may be analyzed. In addition, the object property analysis module 40 may output information related to whether the first object is a moving object according to the property of the first object.

The associated object determination module 50 is a module for identifying whether there is another node connected to the first node including the type of the first object on the knowledge base 85, and identifying information included in the other node. am. That is, the association object determination module may identify whether information on the second object associated with the first object exists based on the information on the first object (eg, the type of the first object, etc.).

Another node connected to the first node including the type of the first object may include the type of the second object associated with (or related to) the type of the first object. The type of the second object (or the object associated with the first object) associated with the type of the first object means the type of the object predicted to appear in the area surrounding the first object. For example, when the type of the first object is a soccer goal post, the type of the second object may be implemented as a soccer ball predicted to appear in the area around the soccer goal post. In this case, the second object may be singular, but is not limited thereto and may be a plurality of two or more.

That is, the related object determination module 50 includes the type of the first object on the knowledge base 85 by using the type of the first object among the information about the first object obtained through the object recognition module 30 . It is possible to identify the first node that has been In addition, the associated object determination module 50 may identify whether another node connected to the first node identified on the knowledge base 85 exists. If it is identified that another node (eg, a second node) connected to the first node exists, the associated object determination module 50 may obtain (or identify) the type of the second object included in the second node. there is.

In addition, the related object determination module 50 may obtain information on the relationship between the type of the first object and the type of the second object by using the knowledge base 85 . For example, an edge connecting the first node and the second node to each other on the knowledge base 85 may include information on a relationship between the type of the first object and the type of the second object.

The information about the relationship may include information related to various relationships between the type of the first object and the type of the second object. For example, the information on the relationship may include information on a directional relationship between the type of the first object and the type of the second object. The information on the direction relationship may include direction information corresponding to a location where the second object appears or has a high probability of being present based on the current location of the first object, but is not limited thereto.

The second image monitoring module 60, the second camera, based on the information about the type of the second object and the relationship between the type of the first object and the type of the second object obtained through the associated object determination module 50 This module determines whether a second ROI among the second images acquired through 120 is determined and monitors the second ROI to identify whether a second object appears in the second ROI. The second region of interest refers to a region determined (or set) to monitor whether a second object appears in the second image.

The second image monitoring module 60 determines, as the second region of interest, an area in the second image where the second object is expected to appear, based on information on the relationship between the type of the first object and the type of the second object. can For example, when information on a relationship including information that a second object is highly likely to appear in a region corresponding to the left direction of the first object is obtained through the associated object determination module 50, the second image monitoring module (60) may determine a region corresponding to the left direction of the first object based on the current position of the first object as the second region of interest. In this case, the second region of interest may be set on a region where the first image and the second image do not overlap, but is not limited thereto.

The second image monitoring module 60 determines a second ROI on the second image, and information on the second ROI (eg, feature information on an image included in the second ROI (eg, , chromaticity, luminance, etc. of pixels constituting the image) and information on the size and shape of an indicator indicating the second region of interest may be acquired.

For example, the second image monitoring module 60 may determine a second region of interest, analyze the image included in the second region of interest, and obtain characteristic information on the image included in the second region of interest. Here, the indicator indicating the second region of interest may have a preset size or shape, but is not limited thereto and may be changed by a user.

The second image monitoring module 60 may identify whether the second object appears on the second ROI. For example, the second image monitoring module 60 may input feature information about an image included in the second ROI and a type of the second object among the information on the second ROI to the second model 93 . Information on whether an object corresponding to the type of the second object appears in the second ROI may be acquired.

As another example, the second image monitoring module 60 identifies information related to the type of the second object from among the first database that collects features related to the types of objects included in the plurality of nodes on the knowledge base 85 . can The second image monitoring module 60 may identify whether the second object appears in the second ROI by monitoring whether information related to the type of the identified second object from among the second ROIs is extracted.

The high-speed imaging performing module 70 controls to perform a high-speed imaging operation of the first camera 110 when information indicating that the second object appears in the second region of interest is acquired through the second image monitoring module 60 . It is a module. That is, when the high-speed photographing performing module 70 controls the first camera 110 to perform the high-speed photographing operation, it means that the SSM function is started.

The memory 130 may store information necessary for the first model 80 and the second model 93 trained to recognize an object to perform an object recognition operation.

When information on the first ROI included on the first image is input by the object recognition module 30 , the first model 80 recognizes an object included in the first ROI and attaches the recognized object to the first model 80 . It is an artificial intelligence model trained to output information about Here, the information on the first ROI may include information about an image included in the first ROI (eg, chromaticity and luminance of pixels constituting the image).

The second model 93 provides information on the image included in the second region of interest by the second image monitoring module 60 (eg, chromaticity and luminance of pixels constituting the image included in the second region of interest). It is an artificial intelligence model trained to output information indicating whether an object corresponding to the type of the second object is included in the second region of interest when the type of the second object is input.

For example, when information on an image included in the second ROI and the type of the second object are input, the second model 93 displays a feature related to the type of the second object on the image included in the second ROI. When it is identified whether information is output and characteristic information related to the type of the second object is extracted, it may be learned to output information indicating that an object corresponding to the type of the second object is included on the second ROI.

The first model 80 and the second model 93 may include an artificial neural network trained to output feature information corresponding to the input image. Examples of artificial neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), GAN. There are Generative Adversarial Networks and Deep Q-Networks, and the neural network model in the present disclosure is not limited to the above-described examples, except as otherwise specified.

Meanwhile, although FIG. 1 shows an embodiment in which the first model 80 and the second model 93 are implemented as separate models, the first model 80 and the second model 93 are implemented as the same model. can be

The memory 130 may store information necessary for the third model 95 to perform an operation of determining the region of interest in the first image. The third model 95 is an artificial intelligence model trained to determine a region of interest among the input first images. For example, the third model 95 may be trained to determine a region in which at least one object included in the first image exists as the region of interest.

Also, the memory 130 may store information necessary for the fourth model 97 to perform an operation. The fourth model 97 outputs a type of a second object associated with the type of the first object or a second object associated with the first object when a label corresponding to the type of the first object or an image for the first object is input. It refers to an artificial intelligence model that has been trained to output information that does not exist.

The memory 130 may store a knowledge base 90 including information related to each of a plurality of objects. The knowledge base 85 may include a plurality of nodes including types of a plurality of objects and an edge including information on relationships between the plurality of objects.

In one embodiment, as shown in FIG. 1B , the knowledge base 85 includes a node including types of a plurality of objects and an edge including information on relationships between the plurality of objects. Knowledge graph (knowledge graph) It can be implemented in the form

For example, it is assumed that the type of the first object is a goal post. The knowledge base 85 may include a first node 98 including the type of the first object. On the knowledge base 85, the first node 98 and the second node (a node including an object related to the first object type, such as a soccer ball) 98-1, is an edge containing information about the relationship in the downward direction. (99-1) can be connected. And, on the knowledge base 85, information on the relationship between the first node 98 and the third node (a node including an object related to the first object type called a basketball) 98-2 in the left/right direction may be connected to an edge 99-2 including . At this time, the fact that the first node 98 and the second node 98-1 are connected to the edge 99-1 including information on the downward relationship is included in the second node 98-1. This may mean that the soccer ball, which is the type of object, is relatively highly likely to be located in the region below the goalpost, which is the type of the first object included in the first node 98 .

The knowledge base 85 may be structured in the form of a knowledge graph as shown in FIG. 1B , but this is only an exemplary embodiment, and the knowledge base 85 is structured in various information base forms to obtain information on the relationship between each object. can be implemented.

Meanwhile, the memory 130 may include a working memory to which the processor 140 can access and write, process, or modify information, and a storage memory for storing and preserving information.

Working memory includes volatile random access memory (RAM), such as dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM), and non-volatile RAM, such as magnetoresistive RAM (MRAM) or phase-change RAM (PRAM). may include at least one of Storage memory includes various types of ROM (eg, one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM)), magnetic storage device (eg, hard disk, etc.) or various auxiliary devices. It may include at least one of a memory device and the like.

Data for performing various operations of the modules 20 , 30 , 40 , 50 , 60 , and 70 for determining when the first camera 110 performs the SSM function may be stored in a storage memory. In addition, data and knowledge base 85 necessary for operating the first model 80 , the second model 93 , the third model 95 , and the fourth model 97 may be stored in the storage memory.

Meanwhile, the working memory may be implemented in a form included in the processor 140 as a component of the processor 140 , but this is only an example, and the working memory is a separate component distinct from the processor 140 . Of course, it can be implemented as

The processor 140 may be electrically connected to the memory 130 to control overall functions and operations of the electronic device 100 . The processor 140 may include one or a plurality of processors to control the operation of the electronic device 100 .

The processor 140 includes the modules 20,30,40,50,60,70 and modules 20,30,40,50,60,70 for determining when to perform the SSM function stored in the storage memory. Data for performing these various operations may be loaded into the working memory. In addition, the processor 140 transfers the first model 80 , the second model 93 , the third model 95 , the fourth model 97 and data necessary for each model to execute various operations to the working memory. can be loaded. Then, the processor 140 may load the knowledge base 85 into the working memory. The processor 140 may perform various operations through various modules and artificial intelligence models based on data loaded into the working memory. Here, the loading refers to an operation of loading and storing data stored in the storage memory into the working memory so that the processor 140 can access it.

On the other hand, the processor 140 may load the data required for each of the various modules 20, 30, 40, 50, 60, 70 and the various artificial intelligence models 80 and 90 to execute the operation at a time, However, the present invention is not limited thereto. The processor 140 may load data necessary for each module and model to execute a specific operation regardless of the order.

Also, when one of various modules or models intends to execute a specific operation, the processor 140 may load only data corresponding to one of the modules or models to perform the specific operation. In addition, when the operation of the various modules (20, 30, 40, 50, 60, 70) and various artificial intelligence models (80, 90) is completed, data necessary to operate the completed module and model are stored in the working memory. can be deleted.

In addition, the processor 140 may load the knowledge base 85 when the related object determination module 50 is loaded, but this is only an embodiment and the knowledge base 85 may be loaded in any order. .

The processor 140 may acquire information on the first object included in the first image obtained by photographing the surroundings of the electronic device 100 through the first camera 110 . Specifically, the processor 140 acquires a first image by photographing the periphery of the electronic device 100 through the first camera 110, and applies a first model learned to recognize an object to the obtained first image. Information on the first object included in the first image may be obtained.

More specifically, the processor 140 may determine the first ROI on the first image acquired using the first camera 110 through the first ROI determining module 20 . In this case, the processor 140 may control the first image to be displayed on the display 150 .

In an embodiment, the processor 140 inputs the first image to the third model 95 trained to determine the region of interest through the first region of interest determination module 20 to determine whether at least one object is included. The region may be determined as the first region of interest. In another embodiment, the processor 140 may first determine, through the first region of interest determining module 20 , a region in which at least one object exists on the first image using an auto focus method. It can be determined as an area of interest.

As another embodiment, the processor 140 may determine a first ROI as a region to which a user touch is input in the first image. The processor 140 may display the determined first region of interest in the first image as an indicator having a preset size and shape (eg, a rectangle). For example, the processor 140 determines a region to which a user touch is input from among the first images displayed on the display 150 as the first ROI, and displays an indicator indicating the determined first ROI on the display 150 . can be controlled as much as possible.

Then, the processor 140 inputs, through the object recognition module 30 , information on the first region of interest obtained by the first region of interest determining module 20 into the first model 80 to obtain a first interest The first object included in the region may be recognized, and information on the recognized first object may be acquired. The information on the first object may include, for example, the type of the first object, the size of the first object, and information on the amount of position change or the movement direction of the first object based on a plurality of frame images included in the first image. .

When it is identified that information on the second object associated with the first object exists based on the information on the first object, the processor 140 may acquire a second image through the second camera 120 .

Specifically, the processor 140 may identify whether the first object is a moving object based on information about the first object. The processor 140, through the object property analysis module 40, analyzes whether the first object has a dynamic property or a static property based on information about the first object, and identifies whether the first object is a moving object. can For example, the processor 140 may identify the amount of change in the position of the first object on the plurality of image frames included in the first image among the information on the first object through the object property analysis module 40 .

For example, when the amount of change in the position of the first object exceeds the first threshold, the processor 140 may analyze that the first object includes a dynamic property, and identifies the first object as a moving object can do. As another example, when the amount of change in the position of the first object is less than or equal to the first threshold value, the processor 140 may analyze that the first object includes a static property, and call the first object as a non-moving object. can be identified. Here, the first threshold value may be a preset value, but of course it may be changed by a user.

The processor 140 may activate the second camera 120 based on the identification result related to whether the first object is a moving object, and obtain a second image using the activated second camera 120 .

Specifically, if the first object is identified as a non-moving object, the processor 140 is configured to generate a second object associated with a type of the first object based on information about the first object (eg, a type for the first object). It may be identified whether information about the object (eg, the type of the second object) exists.

In an embodiment, the processor 140 uses, through the associated object determination module 50 , information on the type of the first object among the information on the first object, such as a plurality of pieces of information included in the knowledge base 85 . A first node including the first object among the nodes may be identified. In addition, the processor 140 may identify, through the associated object determination module 50 , whether or not there is a second node connected to the first node by an edge on the knowledge base 85 .

For example, as shown in FIG. 1B , the knowledge base 85 may be implemented in the form of a knowledge graph including a node including a plurality of object types and an edge having direction relationship information between the plurality of objects. The processor 140 may identify, through the associated object determination module 50 , whether a node including the first object on the knowledge base 85 is connected to another node through an edge.

When the type of the first object is a goal, the processor 140 may identify the first node 98 including the goal among a plurality of nodes included in the knowledge base 85 through the associated object determination module 50 . there is. In addition, the processor 140 may identify whether the first node 98 is connected to another node through an edge on the knowledge base 85 through the associated object determination module 50 . Specifically, the processor 140 determines that the first node 98 on the knowledge base 85 through the associated entity determination module 50 determines the second node 98-1 through the edges 99-1 and 99-2. And it can be identified that it is connected to the third node (98-2).

In another embodiment, the processor 140, through the associated object determination module 50, a rule-based predefined in a label corresponding to the type of the first object among the information on the first object. based) algorithm may be applied to identify whether a second object associated with the type of the first object exists. The label corresponding to the type of the first object means information indicating the characteristic of the type of the first object. The rule-based algorithm refers to an algorithm written to output an object type associated with the type of the first object when a label corresponding to the type of the first object is input. When the information on the object related to the type of the first object is not included in the rule-based algorithm, the processor 140, through the associated object determination module 50, determines that the object related to the type of the first object does not exist. can be identified as

In another embodiment, the processor 140, through the associated object determination module 50, a bounding box indicating a pixel value and a size of the image of the first object among the information on the first object. ), a pixel value and a size of the second object associated with the first object may be identified. Here, the pixel value means a value representing a color property included in the image of the first object, and the bounding box means a box having a minimum size that can include all the shapes of the first object.

Specifically, the processor 140, through the associated object determination module 50, a pixel value of a second object that can be associated with the first object by using a bounding box indicating a pixel value and a size of the image of the first object. and a bounding box indicating a size. Information on the bounding box indicating the pixel value and size of the second object that may be associated with the bounding box indicating the pixel value and size of the image of the first object may be included in the associated object determination module 50 .

In another embodiment, the processor 140 may, in the associated object determination module 50 , identify whether a second object associated with the type of the first object exists through the fourth model 97 . Specifically, the processor 140 extracts a label corresponding to the type of the first object from among the information on the first object, and inputs the extracted label to the fourth model 97 to determine the value of the second object associated with the first object. Information about the type can be obtained in the form of a label. Thereafter, in the second image obtained from the second camera 120 , it may be identified whether the second object type obtained earlier among object information (Label) obtained through the second model exists.

If it is identified that another node connected to the first node exists on the knowledge base 85 , the processor 140 may activate the second camera 120 . For example, if it is identified that another node connected to the first node exists on the knowledge base 85 , the processor 140 may activate the second camera 120 and activate the second camera 120 . .

The processor 140 may acquire a second image by photographing the surroundings of the electronic device 100 using the activated second camera 120 . When the second camera 120 includes a second lens having a shorter focal length than the first camera 110 , the second image is an image obtained by photographing a wider area than the first image, and thus the first image is included in the second image. may be included. At this time, the processor 140 displays the first image on the display 150 , but since a plurality of image frames constituting the second image are stored in the buffer, the first image and the first image using the stored plurality of image frames It may be identified whether the second object exists in an area that does not overlap the second image. That is, the processor 140 does not display the region where the second image and the first image do not overlap on the display 150, but the region is stored in the buffer, and thus performs various operations based on the frames stored in the buffer. can do.

As another example, when a second lens having a longer focal length than that of the first camera 110 is included in the second camera 120 , the first image is an image obtained by photographing a wider area than the second image, so the first image A second image may be included therein.

The processor 140 may identify whether a second object of a type identified as being included in another node connected to the first node exists on the second image acquired through the second camera 120 . In one embodiment, when information that a second object of a type related to the type of the first object exists in an area located in the first direction of the first object using the knowledge base 85 is obtained, the processor 140 may detect whether the second object exists on the second image by detecting an area corresponding to the first direction of the first object in the second image through the second image monitoring module 60 .

For example, information that a basketball ball, which is a second object of a type related to the type of the first object, may be located in regions located in the left and right directions of the goalpost, which is the type of the first object, using the knowledge base 85 Upon obtaining , the processor 140 detects, through the second image monitoring module 60, regions located in the left and right directions of the goalpost in the second image to determine whether a basketball is present on the second image. can be identified.

However, this is only an embodiment, and the processor 140 may identify whether a second object of a type related to the type of the first object exists on the second image by detecting the second image as a whole. That is, the processor 140 may identify the second object included in the second image based on the information obtained through the knowledge base 85 , but may detect the entire area of the second image to identify the second object. can

If it is identified that the second object is present on the second image, the processor 140, through the second image monitoring module 60, based on the information on the relationship between the type of the first object and the type of the second object , a second region of interest may be determined on the second region.

For example, the processor 140 obtains information on a relationship that a second object of a type related to the type of the first object is located in an area corresponding to the first direction of the first object by using the knowledge base 85 . Assume one case. The processor 140 may estimate the movement direction of the second object based on the first direction. For example, the processor 140 may infer that the second object moves in a direction toward the first object from an area (or a current area of the second object) located in the first direction of the first object. . For example, the processor 140 may infer that the basketball ball moves in a direction toward the goal from a region positioned in the left or right direction of the current goal.

As another example, the processor 140, through the second image monitoring module 60, moves the second object by using the amount of position change or the movement direction of the second object on the plurality of image frames constituting the second image. direction can be guessed. The processor 140 may determine the second ROI in the second image based on the estimated motion direction of the second object through the second image monitoring module 60 . Specifically, the processor 140 may determine, as the second region of interest, a region corresponding to the movement direction of the second object estimated based on the region where the second object is located in the second image. The second region of interest may be one, but may be determined to be two or more.

When it is identified that the second object appears on the second region of interest determined through the second image monitoring module 60 , the processor 140 uses the first camera 110 through the high-speed imaging performing module 70 to perform high-speed imaging. shooting can be done. That is, the processor 140 may determine a point in time when the second object moves and appears on the second ROI as a point in time at which high-speed imaging is performed using the first camera 110 among the SSM functions. The SSM function is a function of acquiring a super slow motion image by shooting the surroundings of the electronic device 100 at high speed using the first camera 110, and 32 times the obtained super slow motion image compared to a general image playback speed (30 fps) Or it means that it includes a function to play 4 times slower than normal slow motion (240fps).

Specifically, the processor 140 may monitor whether the second object appears on the determined second ROI through the second image monitoring module 60 . That is, the processor 140 may monitor the second region of interest to identify whether the second object appears on the second region of interest.

For example, the processor 140 may, through the second image monitoring module 60 , include feature information on an image included in the second region of interest among the information on the second region of interest in the second model 93 and the second model 93 . By inputting the type of the second object, it may be identified whether an object corresponding to the type of the second object appears on the second ROI.

As another example, the processor 140 collects, through the second image monitoring module 60 , features related to the types of objects included in the plurality of nodes on the knowledge base 85 , the second object of the first database. It is possible to identify information related to the type of The processor 140 monitors whether information related to the type of the identified second object from among the second ROIs is extracted through the second image monitoring module 60 to determine whether the second object appears in the second ROI can be identified.

Meanwhile, as another embodiment of the present disclosure, when the first object is identified as a moving object based on information on the first object, the processor 140 determines the moving direction of the first object through the object recognition module 30 . can be guessed Specifically, the processor 140 may estimate the moving direction of the first object by calculating the amount of change or the moving direction of the first object on the plurality of image frames included in the first image.

In addition, the processor 140 may determine the third ROI on the first image based on the direction in which the first object moves. The third region of interest refers to a region set to monitor whether the first object appears in the first image. For example, the processor 140 may determine a third ROI as a region corresponding to a direction in which the first object is estimated to move based on a direction in which the first object is currently located on the first image. In this case, the third region of interest may be one, but may be two or more. When it is identified that the first object appears on the third ROI, the processor 140 may perform high-speed imaging using the first camera 110 . Examples related to this will be described in detail in the following section.

In another embodiment of the present disclosure, if it is identified that there is no other node connected to the first node on the knowledge base 85 , the processor 140, through the object recognition module 30 , the first region of interest It is possible to monitor a state change of the first object included in the . When it is identified that the state of the first object is changed, the processor 140 may perform high-speed imaging using the first camera 110 . For example, when the amount of state change of the first object exceeds the second threshold, the processor 140 may perform high-speed imaging using the first camera 110 . That is, the processor 140 may determine the point at which the state of the first object is identified as the point in time at which high-speed imaging is performed using the first camera 110 . An embodiment related thereto will be described in detail with reference to FIG. 7 .

As another embodiment of the present disclosure, the processor 140 may identify that the first object is not included in the first ROI based on information on the first ROI. For example, when a user touch is input to a region not including the first object among the first images displayed on the display 150, the processor 140 removes the region to which the user touch is not included in the first object. 1 It can be determined as a region of interest. In this case, the processor 140 may identify that the first object is not included in the first ROI.

When it is identified that the first object is not included in the first ROI, the processor 140 may activate the second camera 120 . In addition, the processor 140 monitors, through the second image monitoring module 60 , whether an object exists on a region that does not overlap the first image among the second images acquired using the second camera 120 . can do.

Here, it is assumed that the second image is a live view in which a larger area than the first image is captured. If it is identified that the third object exists in a region of the second image that does not overlap with the first image and it is identified that the third object moves in the direction of the first image, the processor 140 uses the first camera 110 to High-speed shooting can be performed. The movement of the third object in the direction of the first image may mean that the third object moves from a region that does not overlap the first image among the second images to a direction in which the first image and the region where the first image overlap. A related embodiment will be described in detail with reference to FIGS. 4A and 4B .

As another embodiment of the present disclosure, when high-speed imaging is performed using the first camera 110 , the processor 140 may obtain information on a relationship between the first ROI and the second ROI. For example, the information about the relationship between the first ROI and the second ROI may include information about the relative positions of the first ROI and the second ROI, and information about an object included in each of the first ROI and the second ROI. Information on the type may be included.

The processor 140 may determine a point in time when high-speed imaging is to be performed using the first camera 110 , based on the obtained information on the relationship between the first region of interest and the second region of interest.

For example, when high-speed imaging is performed using the first camera 110 , a goalpost is included in the first region of interest, a soccer ball is included in the second region of interest, and a second region of interest is included in the right region of the first region of interest. It is assumed that the area is set. Thereafter, when it is identified that the goalpost is included in the determined first region of interest on the first image acquired through the first camera 110 , the processor 140 determines the relationship between the first region of interest and the second region of interest. A second ROI may be set in a right region of the first ROI based on the information about the ROI, and it may be inferred that a soccer ball may be included in the second ROI. And, when a soccer ball appears in the second region of interest, the processor 140 may perform high-speed imaging using the first camera 110 .

As another embodiment of the present disclosure, it is assumed that the first image is an image obtained through an ultra-wide-angle lens and the second image is an image obtained through a wide-angle lens. In this case, the processor 140 may determine an indicator indicating the outer (or boundary) of the first image as the first ROI. And, when a specific object appears on the indicator indicating the first ROI, the processor 140 may perform high-speed imaging using the first camera 110 .

The display 150 may display various information under the control of the processor 140 . The display 150 may display a first image including a plurality of image frames acquired through the first camera 110 . The display 150 may display (or play) the super slow motion image acquired through the first camera 110 32 times slower than a typical image playback speed (30 fps) or 4 times slower than a typical slow motion (240 fps). . However, this is only an example, and the display 150 may display the super slow motion image at a speed set by the user.

The display 150 may be implemented as a touch screen together with a touch panel, and the display 150 may be implemented as a liquid crystal display panel (LCD), organic light emitting diodes (OLED), etc., in some cases, a flexible display, It is also possible to be implemented as a transparent display or the like. However, it is not limited to the above-described implementation, and the display 150 may be implemented differently depending on the type of the electronic device 100 .

Meanwhile, the function related to artificial intelligence according to the present disclosure is operated through the processor 140 and the memory 130 . The processor 140 may include one or a plurality of processors. In this case, one or more processors include a general-purpose processor such as a central processing unit (CPU), an application processor (AP), and a digital signal processor (DSP), and a graphics-only processor such as a graphic processing unit (GPU) and a vision processing unit (VPU). Alternatively, it may be a processor dedicated to artificial intelligence, such as a Neural Processing Unit (NPU).

One or a plurality of processors 140 control to process input data according to a predefined operation rule or artificial intelligence model stored in the memory 130 . Alternatively, when one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

A predefined action rule or artificial intelligence model is characterized in that it is created through learning. Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created means burden. Such learning may be performed in the device itself on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system.

Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

The artificial intelligence model includes a plurality of artificial neural networks, and the artificial neural network may be composed of a plurality of layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized.

2 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

The electronic device 100 may acquire information on the first object included in the first image obtained by photographing the surroundings of the electronic device through the first camera (S210). Specifically, the electronic device 100 may determine the first ROI on the first image. The process of determining the first ROI on the first image will be described in detail with reference to FIG. 3 .

The electronic device 100 may identify whether the first object is included in the determined first ROI. When it is identified that the first object is included in the first ROI, the electronic device 100 applies the information on the first ROI to the first artificial intelligence model learned to recognize the object to be included in the first ROI. Information on the first object may be obtained. A process in which the electronic device 100 acquires information on the first object will be described in detail with reference to FIG. 3 . Meanwhile, a case in which it is identified that the first object is not included in the first ROI will be described in detail with reference to FIG. 4A .

When it is identified that information on a second object associated with the first object exists based on the obtained information on the first object, the electronic device 100 may acquire a second image through the second camera ( S220 ). .

Specifically, the electronic device 100 may identify whether the first object is a moving object using information on the first object. When the first object is identified as a non-moving object, the electronic device 100 determines whether information about the second object (eg, the type of the second object, etc.) associated with the type of the first object exists on the knowledge base. whether or not it can be identified. When it is identified that information on the type of the second object exists on the knowledge base, the electronic device 100 may activate the second camera. Then, the electronic device 100 may acquire the second image by using the activated second camera. A related embodiment will be described in detail with reference to FIG. 3 .

However, this is only an embodiment, and the electronic device 100 may identify whether the type of the second object associated with the type of the first object exists without using the knowledge base. Since the related embodiments have been described above, redundant descriptions will be omitted.

Meanwhile, a case in which the first object is identified as a moving object will be described in detail with reference to FIG. 6 . In addition, a case in which it is identified that there is no information on the type of the second object associated with the type of the first object on the knowledge base will be described in detail with reference to FIG. 7 .

The electronic device 100 may determine a point in time to perform high-speed imaging using the first camera based on whether the second object is included in the acquired second image (S230). When it is identified that the second object is present on the second image, the electronic device 100 may determine the second ROI based on the movement direction of the second object. The electronic device 100 may monitor whether the second object appears on the determined second ROI. When it is identified that the second object appears on the second ROI, the electronic device 100 may perform high-speed imaging using the first camera. An embodiment related thereto will be described in detail with reference to FIG. 5 .

3 is a flowchart illustrating a process in which the electronic device 100 activates a second camera according to an embodiment of the present disclosure. S310 to S330 of FIG. 3 embody the step S210 of FIG. 2 .

The electronic device 100 may determine the first ROI on the first image acquired through the first camera (S310).

For example, the electronic device 100 may input a first image to a third model trained to determine a region of interest to determine a region including at least one object as the first region of interest. The third model may be an artificial intelligence model trained to determine a region including at least one object included in the first image as a region of interest.

As another example, the electronic device 100 may determine the first ROI on the first image by using an auto-focus method. Auto focus refers to a function of automatically focusing on a specific object included in the first image. When one of the objects included in the first image is focused by the auto-focus method, the electronic device 100 may determine a region including the focused object in the first image as the first ROI.

As another example, the electronic device 100 may determine a region to which a user touch is input among a first image displayed on a display including a touch screen as the first region of interest. The electronic device 100 may display a bounding box having a preset size or shape in a region to which a user touch is input, and set the region in which the bounding box is displayed as the first region of interest.

The electronic device 100 may identify whether the first object is included in the first ROI (S320). Meanwhile, when the third model or the autofocus method is used, since the object is included in the first ROI, the electronic device 100 may omit step S320 and perform step S330.

It is assumed that a first ROI is determined as a region not including the first object as a user touch is input to a region not including the first object among the first images displayed on the display. In this case, the electronic device 100 may identify that the first object is not included in the first ROI. A case in which the first object is not included in the first ROI will be described in detail with reference to FIG. 4A .

When the first object is included in the first ROI, the electronic device 100 inputs information about the first ROI into the first model trained to recognize the object, and the first object included in the first ROI It is possible to obtain information on (S330). The information on the first ROI may include information about an image included in the first ROI (eg, chromaticity and luminance of pixels constituting the image). The information on the first object may include, for example, the type of the first object, the size of the first object, and information on the amount of change in the position of the first object based on a plurality of frame images included in the first image.

The electronic device 100 may identify whether the first object is a moving object by using the information on the first object ( S340 ). For example, the electronic device 100 identifies the amount of change in the position of the first object on the plurality of image frames constituting the first image among the information on the first object, and based on the amount of change in the position of the identified first object, the electronic device 100 1 You can analyze the properties of an object. Also, the electronic device 100 may identify whether the first object is a moving object according to the property of the first object.

When the first object is identified as a non-moving object, the electronic device 100 may identify whether information on the type of the second object associated with the type of the first object exists on the knowledge base ( S350 ). Since the embodiment related to whether information on the type of the second object associated with the type of the first object exists on the knowledge base has been described above, a redundant description will be omitted.

When it is identified that information on the type of the second object associated with the type of the first object exists on the knowledge base, the electronic device 100 may activate the second camera. An embodiment after the second camera is activated will be described in detail with reference to FIG. 5 . Meanwhile, a case in which the first object is a moving object will be described in detail with reference to FIG. 6 . In addition, a case in which information on the type of the second object associated with the type of the first object does not exist on the knowledge base will be described in detail with reference to FIG. 6 .

4A is a flowchart illustrating a process in which the electronic device 100 performs high-speed imaging using a first camera, according to an embodiment of the present disclosure. Referring to FIG. 4B , FIG. 4A is a case in which it is identified (S320-N) that the first object is not included in the first region of interest (ROI) 407 on the first image 400 . It is a flowchart for explaining an example in detail.

When it is identified that the first object is not included in the first ROI, the electronic device 100 may activate the second camera (S410). The electronic device 100 may acquire the second image by using the activated second camera. Here, it is assumed that the first image 400 is a wide-view and the second image 405 is an ultra-wide view having a wider field of view than the first image 400 .

The electronic device 100 may monitor whether an object exists on a region that does not overlap the first image 400 among the second images 405 acquired using the second camera ( S420 ).

If it is identified that the third object exists on the area where the first image 400 does not overlap among the second images 405 and it is identified that the third object moves in the direction of the first image, the electronic device 100 displays the second image 1 High-speed shooting may be performed using the camera 110 (S430). In this case, the movement of the third object in the direction of the first image means that the third object does not overlap the first image among the second images in the direction of the area where the first image and the second image overlap (eg, arrow 411-1). to 411-8) direction).

Meanwhile, as another embodiment of the present disclosure, the electronic device 100 may set a second region of interest (or sub-ROI) 409 in a region that does not overlap the first image among the second images. For example, as shown in FIG. 4B , the electronic device 100 may set the second region of interest 409 in a region overlapping the first image and the second image. In addition, the electronic device 100 may monitor whether a specific object appears in the second ROI 409 .

For example, if it is identified that the third object appears on the second region of interest 409 while moving in the direction of the arrow 411-4, the electronic device 100 may perform high-speed imaging using the first camera. there is. The electronic device 100 may perform the SSM function while performing high-speed photographing using the first camera.

FIG. 5 is a flowchart illustrating a process of determining a time point at which the electronic device 100 performs high-speed imaging using a first camera according to another embodiment of the present disclosure, and operations after S360 are described.

The electronic device 100 may acquire a second image by using the activated second camera (S510). The electronic device 100 may identify whether the second object is present on the second image (S520). As an embodiment, the electronic device 100 may identify whether the second object exists on the second image based on information included in the knowledge base. For example, the electronic device 100 provides information on the relationship between the type of the first object and the type of the second object included in the knowledge base (eg, a direction between the type of the first object and the type of the second object). relationship information, etc.), it is possible to identify whether the second object exists on the second image.

When it is identified that the second object is present on the second image, the electronic device 100 may estimate the movement direction of the second object ( S530 ). In one embodiment, information that a second object of a type related to the type of the first object is located in an area corresponding to the first direction of the first object on the knowledge base 85 (that is, information about the direction relationship) Assume that it is included. The electronic device 100 may estimate the movement direction of the second object based on the first direction. Specifically, the electronic device 100 may estimate that the second object moves in a direction from the current location (eg, an area positioned in the first direction of the first object) toward the first object.

As another embodiment, the electronic device 100 may estimate the direction in which the second object will move by calculating the amount of change in position or the direction of movement of the second object on a plurality of frames constituting the second image. For example, if it is calculated that the second object moves by the first position change amount in the direction toward the first object on the plurality of frames constituting the second image, the electronic device 100 determines that the second object moves toward the first object. It can be assumed that the heading "?toward?* moves."

Meanwhile, when it is identified that the second object does not exist on the second image, the electronic device 100 may estimate the moving direction of the first object. An embodiment related thereto will be described in detail with reference to FIG. 6 .

The electronic device 100 may determine a second ROI on the second image based on the movement direction of the second object ( S540 ). Here, the second region of interest refers to a region determined (or set) to monitor whether a second object appears in the second image. The electronic device 100 may determine a region corresponding to the movement direction of the second object estimated based on the region in which the second object is located in the second image as the second region of interest. The second region of interest may be one, but may be determined to be two or more.

The electronic device 100 may monitor whether the second object appears on the second ROI (S550). As an embodiment, the electronic device 100 inputs feature information about an image included in the second ROI and a type of the second object among the information on the second ROI of the second model 93 to receive the second ROI. Information on whether an object corresponding to the type of the second object appears in the area may be acquired.

As another example, the electronic device 100 may identify information related to the type of the second object from among the first database in which features related to the types of objects included in the plurality of nodes on the knowledge base are collected. In addition, the electronic device 100 may identify whether the second object appears in the second ROI by monitoring whether information related to the type of the identified second object from among the second ROIs is extracted.

When it is identified that the second object appears on the second ROI, the electronic device 100 may perform high-speed imaging using the first camera (S560). That is, the electronic device 100 may determine a point in time at which the second object appears on the second ROI while moving as a point in time at which high-speed imaging is performed using the first camera among the SSM functions.

6 is a flowchart illustrating a process of determining a time point at which an electronic device performs high-speed imaging using a first camera, according to an embodiment of the present disclosure.

When the first object is identified as a moving object (S340-Y), the electronic device 100 may perform step S610 of FIG. 6 . The electronic device 100 may estimate the moving direction of the first object (S610). Specifically, the electronic device 100 may estimate the moving direction of the first object by calculating the amount of change or the moving direction of the first object on the plurality of image frames included in the first image.

The electronic device 100 may determine a third ROI on the first image based on the estimated direction in which the first object moves ( S620 ). The third region of interest refers to a region set to monitor whether the first object appears in the first image. For example, the electronic device 100 may determine a third ROI as a region corresponding to a direction in which the first object is estimated to move based on a direction in which the first object is currently located on the first image. In this case, the third region of interest may be one, but may be two or more.

The electronic device 100 may monitor whether the first object appears on the third ROI (S630). When it is identified that the first object appears on the third ROI, the electronic device 100 may perform high-speed imaging using the first camera (S640). That is, when the first object included in the first image is a moving object, the electronic device 100 determines the first ROI on the first image, and based on whether the first object appears in the first ROI Thus, high-speed imaging can be performed using the first camera.

7 is a flowchart illustrating a process of determining a point in time when the electronic device 100 performs high-speed imaging using a first camera, according to an embodiment of the present disclosure. When information on the type of the second object associated with the type of the first object does not exist on the knowledge base (S350-N) or when the second object does not exist on the second image (S520-N), the former The device 100 may monitor a state change of the first object ( S710 ).

The electronic device 100 may identify whether the amount of state change of the first object exceeds the second threshold ( S720 ). Here, the state change amount refers to the degree to which a physical quantity, such as a size, volume, shape, etc. of the first object is changed numerically. In addition, the second threshold value may be a preset value calculated by experiments or research, but is not limited thereto, and may be changed by a user.

In one embodiment, it is assumed that the first object is implemented as a balloon filled with water. When the balloon filled with water bursts, the shape or volume of the balloon filled with water may be significantly changed. In this case, the electronic device 100 may identify that the amount of change in the state of the balloon filled with water exceeds the second threshold.

When it is identified that the amount of change in the state of the first object exceeds the second threshold, the electronic device 100 may perform high-speed imaging using the first camera ( S730 ). That is, the electronic device 100 may determine a point in time when the state of the first object is changed as a point in time of high-speed photographing using the first camera.

As another embodiment of the present disclosure, the electronic device 100 may include information of an image representing the first object among states of the first object (eg, color, contrast, pixel value, etc. of the image of the first object) When changed, high-speed photographing may be performed using the first camera. For example, when the amount of change in the information of the image representing the first object (eg, the amount of change in the color of the image of the first object, etc.) exceeds the third threshold, the electronic device 100 controls the first camera. It can be used to perform high-speed photography.

8 is a block diagram illustrating in detail the configuration of the electronic device 100 according to another embodiment of the present disclosure. 8 , the electronic device 100 includes a first camera 110 , a second camera 120 , a memory 130 , a processor 140 , a display 150 , a communication unit 160 , and a speaker ( 170 ), a microphone 180 , and an input unit 190 . Since the first camera 110 , the second camera 120 , the memory 130 , the processor 140 , and the display 150 have been described in detail with reference to FIG. 1A , redundant descriptions will be omitted.

The communication unit 160 includes a circuit and may communicate with an external device. In this case, the communication connection of the communication unit 160 with the external device may include communication through a third device (eg, a repeater, a hub, an access point, a server or a gateway, etc.).

The communication unit 160 may include various communication modules to communicate with an external device. For example, the communication unit 120 may include a wireless communication module, for example, 5G (5TH Generation), LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA) It may include a cellular communication module using at least one of , , and the like.

As another example, the wireless communication module may include, for example, at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), Zigbee, radio frequency (RF), or body area network (BAN). can However, this is only an embodiment and the communication unit 120 may include a wired communication module.

The first model 80 , the knowledge base 85 , the second model 93 , the third model 95 , and the fourth model 97 shown in FIG. 1 may be included in an external server. The communication unit 160 may receive the knowledge base 85 from an external server.

In an embodiment, the communication unit 160 may transmit information on the first ROI to an external server including the first model 80 , and may receive information on the first object from the external server.

In another embodiment, the communication unit 160 transmits information about the image included in the second region of interest and the type of the second object to an external server including the second model 93, and from the external server to the second interest region. Information indicating whether an object corresponding to the type of the second object is included in the region may be received.

In another embodiment, the communication unit 160 transmits the first image to an external server including the third model 95, and receives information on a region set as the first region of interest among the first images input from the external server. can receive The processor 140 may determine the first ROI from among the first images based on the information on the region set as the first ROI obtained through the communication unit 160 .

In another embodiment, the communication unit 160 transmits a label corresponding to the type of the first object or an image for the first object to an external server including the fourth model 97, and transmits the image of the first object from the external server. It may receive information that the type of the second object associated with the type or the second object associated with the first object does not exist.

The speaker 170 is configured to output various audio data on which various processing operations such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown). Also, the speaker 170 may output various notification sounds or voice messages.

For example, the speaker 170 may output a notification sound notifying the time when high-speed imaging is performed using the first camera 110 . As another example, the speaker 160 may output audio data included in the SSM image.

The microphone 180 is configured to receive a voice input from a user. The microphone 180 may be provided inside the electronic device 100 , but may be provided outside and electrically connected to the electronic device 100 . In addition, when the microphone 180 is provided outside, the microphone 180 may transmit a user voice signal generated through a wired/wireless interface (eg, Wi-Fi, Bluetooth) to the processor 140 .

The microphone 180 is a wake-up word (or a trigger word) capable of waking up a dialog system (not shown), which is an artificial intelligence model that can recognize and understand the voice input from the user. (trigger word)) can be input. In addition, the microphone 180 may receive a user voice including a command to perform high-speed photographing using the first camera 110 . The processor 140 may input a user's voice including a command to perform high-speed imaging acquired through the microphone 180 into the conversation system. In addition, the processor 140 may perform high-speed photographing using the first camera 110 based on information obtained through the conversation system.

The input unit 190 includes a circuit and may receive a user input for controlling the electronic device 100 . In particular, the input unit 190 may include a touch panel for receiving a user touch input using a user's hand or a stylus pen, a button for receiving a user manipulation, and the like. As another example, the input unit 190 may be implemented as another input device (eg, a keyboard, a mouse, a motion input unit, etc.). Meanwhile, the input unit 170 may receive various user commands.

On the other hand, the drawings accompanying the present disclosure are not intended to limit the technology described in the present disclosure to specific embodiments, and various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure are provided. should be understood as including In connection with the description of the drawings, like reference numerals may be used for like components.

In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this disclosure, expressions such as "A or B," "at least one of A and/and B," or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used in the present disclosure, expressions such as “first,” “second,” “first,” or “second,” may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression “configured to (or configured to)” as used in this disclosure, depending on the context, for example, “suitable for,” “having the capacity to” ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a coprocessor configured (or configured to perform) A, B, and C" may include a processor dedicated to performing the operations (eg, an embedded processor), or executing one or more software programs stored in a memory device. By doing so, it may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). As an apparatus capable of calling and operating according to the called instruction, it may include the server cloud according to the disclosed embodiments. When the instruction is executed by a processor, the processor directly or under the control of the processor performs other components A function corresponding to the above command can be performed by using it.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the 'non-transitory storage medium' does not include a signal and only means that it is tangible and does not distinguish that data is semi-permanently or temporarily stored in the storage medium. For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, the method according to various embodiments disclosed in the present disclosure may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable app) is at least temporarily stored in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server, or is temporarily stored can be created with

Each of the components (eg, a module or a program) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be various It may be further included in the embodiment. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity, so that functions performed by each corresponding component prior to integration may be performed identically or similarly. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallelly, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

100: electronic device 110: first camera
120: second camera 130: memory
140: processor 150: display

Claims (20)

In an electronic device,
a first camera including a first lens having a first focal length;
a second camera including a second lens having a second focal length different from the first focal length;
Memory; and
obtaining information on a first object included in a first image obtained by photographing a periphery of the electronic device through the first camera;
When information on a second object associated with the first object is identified based on the obtained information on the first object, a second image is obtained through the second camera,
and a processor configured to determine when to perform high-speed imaging using the first camera, based on whether the second object is included in the acquired second image. According to claim 1,
The processor is
determining a first region of interest on the obtained first image,
The electronic device obtains information on the first object included in the first ROI by applying the information on the first ROI to a first model trained to recognize the object. According to claim 1,
The memory stores a knowledge base including information related to each of a plurality of objects,
The processor is
Identifies whether the first object is a moving object by using the information on the first object,
If the first object is identified as a non-moving object, it is identified whether information on the type of the second object associated with the type of the first object exists on the knowledge base,
When it is identified that the information on the type of the second object exists on the knowledge base, the electronic device acquires the second image by activating the second camera. 4. The method of claim 3,
The knowledge base includes a knowledge base including a plurality of nodes including types of the plurality of objects and an edge including information on directional relationships between the plurality of nodes,
The processor is
A first node including the type of the first object is identified among the plurality of objects included in the knowledge base, and the identified first node and the identified first node through an edge including information on the direction relationship on the knowledge base An electronic device that identifies whether a connected node exists. 5. The method of claim 4,
The processor is
When it is identified on the knowledge base that there is a second node connected to the first node, activating the second camera to acquire the second image;
An electronic device for identifying whether the second object exists on the second image based on relationship information included in an edge connected between the first node and the second node. 6. The method of claim 5,
The processor is
If it is identified that the second object is present on the second image, the direction of movement of the second object is estimated,
The electronic device determines a second ROI on the second image based on the estimated motion direction of the second object. 7. The method of claim 6,
The processor is
Identifies whether the second object appears on the determined second region of interest,
The electronic device determines a time point at which the second object appears on the second region of interest as a time point at which high-speed imaging is performed using the first camera. 4. The method of claim 3,
The processor is
If the first object is identified as a moving object using information about the first object, the direction in which the first object moves is estimated,
determining a third ROI on the first image based on the estimated moving direction of the first object,
The electronic device determines a point in time at which the first object is identified as appearing on the third region of interest as a point in time at which high-speed imaging is performed using the first camera. 4. The method of claim 3,
The processor is
If it is identified that there is no information on the type of the second object associated with the type of the first object on the knowledge base, monitoring a change in the state of the first object included in the first ROI;
An electronic device that determines a point in time at which it is identified that the state of the first object changes as a point in time at which high-speed imaging is performed using the first camera. 3. The method of claim 2,
The processor is
If it is identified that the first object is not included in the first region of interest based on the information on the first region of interest, activating the second camera to acquire the second image;
identifying a third object on the second image that does not overlap with the first image,
The electronic device determines a point in time at which the third object is identified as moving in the first image direction as a point in time at which high-speed imaging is performed using the first camera. A method of controlling an electronic device, comprising: a first camera including a first lens having a first focal length; and a second camera including a second lens having a second focal length different from the first focal length;
obtaining information on a first object included in a first image obtained by photographing a periphery of the electronic device through the first camera;
acquiring a second image through the second camera when information on a second object associated with the first object is identified based on the obtained information on the first object; and
and determining a point in time when high-speed imaging is to be performed using the first camera, based on whether the second object is included in the acquired second image. 12. The method of claim 11,
The step of obtaining information about the first object includes:
determining a first region of interest on the acquired first image; and
and acquiring information on the first object included in the first ROI by applying the information on the first ROI to a first model trained to recognize an object. 12. The method of claim 11,
Acquiring the second image comprises:
identifying whether the first object is a moving object using information about the first object;
identifying whether information on a type of a second object associated with the type of the first object exists on a knowledge base included in a memory of the electronic device when the first object is identified as a stationary object; and
and acquiring the second image by activating the second camera when it is identified that the information on the type of the second object exists on the knowledge base. 14. The method of claim 13,
The knowledge base includes a knowledge base including a plurality of nodes including types of the plurality of objects and an edge including information on directional relationships between the plurality of nodes,
The determining step is
A first node including the type of the first object is identified among the plurality of objects included in the knowledge base, and the identified first node and the identified first node through an edge including information on the direction relationship on the knowledge base A control method comprising a; identifying whether a connected node exists. 15. The method of claim 14,
The determining step is
acquiring the second image by activating the second camera when it is identified that a second node connected to the first node exists on the knowledge base; and
and identifying whether the second object exists on the second image based on relationship information included in an edge connected between the first node and the second node. 16. The method of claim 15,
The determining step is
estimating a movement direction of the second object when it is identified that the second object is present on the second image; and
and determining a second ROI on the second image based on the estimated motion direction of the second object. 17. The method of claim 16,
The step of determining when to perform the high-speed shooting includes:
identifying whether the second object appears on the determined second region of interest; and
and determining a point in time at which the second object appears on the second region of interest as a point in time at which high-speed imaging is performed using the first camera. 14. The method of claim 13,
The determining step is
estimating a moving direction of the first object when the first object is identified as a moving object using information about the first object;
determining a third ROI on the first image based on the estimated moving direction of the first object; and
and determining a point in time at which the first object is identified as appearing on the third region of interest as a point in time at which high-speed imaging is performed using the first camera. 14. The method of claim 13,
The determining step is
monitoring a state change of the first object included in a first ROI when it is identified that information on the type of the second object associated with the type of the first object does not exist on the knowledge base; and
and determining a point in time when it is identified that the state of the first object changes as a point in time at which high-speed imaging is performed using the first camera. 13. The method of claim 12,
The determining step is
acquiring the second image by activating the second camera when it is identified that the first object is not included in the first ROI based on the information on the first ROI;
identifying a third object on the second image that does not overlap with the first image; and
and determining a point in time at which the third object is identified as moving in the first image direction as a point in time at which high-speed imaging is performed using the first camera.

Images