KR20230114567A - Display apparatus and method for controlling display apparatus - Google Patents

Abstract

An electronic device and a method for controlling the electronic device are provided. The electronic device may include a plurality of cameras; display; Memory; and identifying one of the plurality of cameras based on a first location of the electronic device and a second location set based on the first location, the identified camera while the electronic device is in the first location. Obtaining a first object image by photographing an object, identifying a transformation function corresponding to the second position based on the 3D image of the object and the second position, and identifying the first object image and the second position. and a processor for obtaining a second object image corresponding to the second position based on a conversion function and displaying the first object image and the second object image on the display.

Description

Electronic device and control method of electronic device {DISPLAY APPARATUS AND METHOD FOR CONTROLLING DISPLAY APPARATUS}

The present disclosure relates to an electronic device, and more particularly, to an electronic device for displaying a plurality of images of an object that can be obtained at a different point of time from a point of time when the camera captures the object based on an image of the object captured through a camera. It's about the device.

As the mobile industry develops, technology related to camera modules embedded in mobile devices has also developed. In particular, since 2010, when smart phones were widely distributed, despite the fact that the growth rate of the smart phone market has gradually slowed down, technology related to camera modules has been continuously developed. For this reason, each company that produces smart phones is striving to differentiate their smart phones through technological differentiation of camera modules mounted on smart phones as the final step for differentiating their smart phones. For example, various sensors such as a Time of Flying (TOF) sensor and a Lidar sensor are included in the camera module so that the user can acquire various information about the object through the camera module. In addition, by moving away from the conventional smart phone that included only one camera module on the front and back, various lenses such as wide-angle lens, standard lens, and telephoto lens are included in the camera module according to the angle of view, so that the user can view the subject with only one smartphone. A variety of images were obtained.

However, in spite of the development of the camera module, it is still possible to acquire an image of a subject through the camera module only at a fixed location of the smart phone. That is, it is still impossible to simultaneously acquire images of a subject from a plurality of viewpoints. Accordingly, users have to repeat the process of changing the position of the smart phone and reviewing the image of the subject captured at the changed position in order to find the optimal angle for photographing the subject. In particular, in order to shoot a movie that requires shooting images from various viewpoints for one scene, a user must prepare a plurality of smart phones or additional equipment such as a camera each time.

Therefore, there is a need for a device capable of providing images of a subject from a plurality of viewpoints using only one camera module without a plurality of devices. This is because if the user can simultaneously receive images captured from multiple viewpoints of the subject using only one smart phone, there is no need to repeat the above-described process or to prepare additional equipment.

The present disclosure has been made to solve the above-described problems, and an object of the present disclosure is to provide an electronic device and a control method for providing captured images of an object from a plurality of viewpoints without a plurality of camera modules.

An electronic device according to an aspect of the present disclosure for achieving the above object is provided based on a plurality of cameras, a display, a memory, and a first location of the electronic device and a second location set based on the first location. One of the cameras is identified, and a first object image is obtained by capturing an object through the identified camera while the electronic device is in the first position, and a 3D image of the object and the second position are obtained. A transformation function corresponding to the second position is identified based on, a second object image corresponding to the second position is obtained based on the first object image and the transformation function, and the first object image and and a processor displaying the second object image on the display.

Also, the processor identifies an angle of view of each of the plurality of cameras, and identifies a camera corresponding to an angle of view including the second position among the plurality of cameras as the one camera.

In addition, the processor captures the object from a plurality of viewpoints through the camera, obtains a plurality of object images corresponding to the plurality of viewpoints, and based on an object identified in each of the plurality of object images, Acquire a 3D image of an object.

In addition, the processor obtains coordinate values corresponding to feature points of objects identified at each of a plurality of viewpoints based on the 3D image, and generates a plurality of transformation matrices for conversion between the obtained plurality of coordinate values. Obtained, and based on a feature point of the object identified in the first object image and a transformation matrix corresponding to the feature point of the object identified in the first object image among the obtained plurality of transformation matrices, at the second position A corresponding second object image is acquired.

In addition, the memory includes a neural network model learned to generate a 3D image of an object based on an input object image, and the processor inputs the first object image to the neural network model to generate a 3D image of the object. A 3D image of the object is obtained, and a second object image corresponding to the second location is obtained based on the 3D image obtained through the neural network model.

In addition, the processor identifies a reference range for the object, and if the second location is included in the identified reference range, obtains the second object image based on the first object image and a transformation matrix; When the second position is not included in the identified reference range, the second object image is obtained based on the 3D image.

Also, the processor identifies the reference range based on the identified information on the view angle of the camera.

In addition, when a third object image is acquired by photographing the object at a location different from the first location through the camera, the processor may include an object not included in the first object image but included in the second object image. It identifies whether at least a part of is included in the third object image, and at least a part of the object is included in the third object image and at least a part of the object included in the third object image is the second object. If different from the image, the 3D image is updated based on the third object image.

Also, the processor obtains a second object image corresponding to the second location based on the updated 3D image, and displays the second object image on the display.

In the method for controlling an electronic device including a plurality of cameras according to another aspect of the present disclosure for achieving the above object, based on a first position of the electronic device and a second position set based on the first position Identifying one camera among the plurality of cameras, acquiring a first object image by photographing an object through the identified camera while the electronic device is in the first location, and obtaining a 3D image of the object. and identifying a transform function corresponding to the second position based on the second position, obtaining a second object image corresponding to the second position based on the first object image and the transform function. and displaying the first object image and the second object image on a display.

The identifying of the one camera may include identifying an angle of view of each of the plurality of cameras and identifying a camera corresponding to an angle of view including the second position among the plurality of cameras as the one camera. includes

In addition, the step of acquiring a plurality of object images corresponding to the plurality of viewpoints by photographing the object from a plurality of viewpoints through the camera, and based on the object identified in each of the plurality of object images, information about the object is obtained. Acquiring a 3D image.

In addition, the step of identifying the conversion function may include obtaining coordinate values corresponding to feature points of the object identified at each of a plurality of viewpoints based on the 3D image, and converting between the obtained plurality of coordinate values. obtaining a plurality of transformation matrices for the first object image, wherein the obtaining of the second object image includes a feature point of an object identified in the first object image and the first object image of the obtained plurality of transformation matrices. and obtaining a second object image corresponding to the second position based on a transformation matrix corresponding to the feature point of the object identified in .

The method may further include acquiring a 3D image of the object by inputting the first object image to a neural network model, and the acquiring of the second object image may include the step of obtaining a 3D image of the object through the neural network model. A second object image corresponding to the second position is obtained based on the dimensional image.

The obtaining of the second object image may include identifying a reference range for the object, and if the second location is included in the identified reference range, the second location based on the first object image and a transformation matrix. A 2-object image is acquired, and if the second location is not included in the identified reference range, the second object image is acquired based on the 3-dimensional image.

Also, the reference range is identified based on the identified information on the angle of view of the camera.

In addition, acquiring a third object image by photographing the object at a location different from the first location through the camera, at least a part of the object included in the second object image but not included in the first object image identifying whether is included in the third object image, and at least part of the object is included in the third object image and at least part of the object included in the third object image is identical to the second object image. In another case, updating the 3D image based on the third object image.

In the displaying of the second object image, a second object image corresponding to the second location is acquired based on the updated 3D image, and the second object image is displayed on the display.

Other specific details of the invention are included in the detailed description and drawings.

According to the present disclosure described above, without a plurality of cameras or a plurality of electronic devices including the cameras, the user can view the object photographed through the camera in real time at a plurality of positions different from the photographing position of the camera. Images can be provided.

Through this, the user can intuitively observe captured images in a plurality of positions of the corresponding object, and may determine an optimal capturing angle for capturing the corresponding object.

In addition, iteratively compares an image of the corresponding object at a plurality of locations that has already been output and stored in memory or is scheduled to be output and an actual captured image, and updates the image of the corresponding object at a plurality of locations through the actually captured image. By doing so, the user can be provided with a plurality of more accurate images of the corresponding object.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary view for explaining an electronic device according to an embodiment of the present disclosure.
2 is a schematic configuration diagram of an electronic device according to an embodiment of the present disclosure.
3 is a schematic flowchart of a method for controlling an electronic device according to an embodiment of the present disclosure.
4A is an exemplary diagram for explaining relatively identifying a second location based on a first location according to an embodiment of the present disclosure.
4B is an exemplary diagram for explaining identification of a first location and a second location according to another embodiment of the present disclosure.
5 is a flowchart schematically illustrating a method of identifying a transform function corresponding to a second position based on a 3D image of an object, according to an embodiment of the present disclosure.
6 is a schematic diagram of a control method of an electronic device for obtaining a transformation matrix based on coordinate values of feature points of an object and acquiring a second object image using the transformation matrix, according to an embodiment of the present disclosure; is a flow chart.
7 is an exemplary view illustrating obtaining a transformation matrix for transformation using coordinate values corresponding to feature points of an object according to an embodiment of the present disclosure.
8 is an exemplary view illustrating obtaining an object image corresponding to a second position based on a transformation matrix according to an embodiment of the present disclosure.
9 is an exemplary view illustrating obtaining a second object image using a 3D image of the object according to an embodiment of the present disclosure.
10 is a diagram for explaining setting a reference range based on view angle information set in a camera and generating a second object image according to the reference range, according to an embodiment of the present disclosure.
11 is an exemplary view of displaying a first object image and a second object image on a display according to an embodiment of the present disclosure.
12 is a flowchart schematically illustrating a control method of an electronic device for updating a 3D image of an object based on an image of the object acquired at a second location according to an embodiment of the present disclosure.
13A and 13B are exemplary diagrams illustrating displaying a second object image 30 based on an updated 3D image.
14 is a detailed configuration diagram of an electronic device according to an embodiment of the present disclosure.

The terms used in the embodiments of the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

In the present disclosure, expressions such as “has,” “can have,” “includes,” or “can include” indicate the presence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

The expression at least one of A and/or B should be understood to denote either "A" or "B" or "A and B".

Expressions such as "first," "second," "first," or "second," used in the present disclosure may modify various elements regardless of order and/or importance, and may refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that an element may be directly connected to another element, or may be connected through another element (eg, a third element).

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module and implemented by at least one processor (not shown), except for "modules" or "units" that need to be implemented with specific hardware. It can be.

1 is an exemplary diagram for explaining an electronic device according to an embodiment of the present disclosure.

The electronic device 100 according to an embodiment of the present disclosure displays an image of an object photographed by the camera 110 on the display 120 . The electronic device 100 displays on the display 120 a captured image obtained when the electronic device 100 is positioned looking at an object through the camera 110 . However, according to an embodiment of the present disclosure, the electronic device 100, in addition to the photographed image of the object obtained through the camera 110, the image obtained when the object is photographed at a location other than the location where the electronic device is located. provides more to the user.

Specifically, referring to FIG. 1 , it is assumed that the current electronic device 100 captures an object 200 at location A 11 using the camera 110 . In this case, according to an embodiment of the present disclosure, the electronic device 100 has virtual electronic devices 100-1 and 100-2 at location B 12 or location C 13 other than location A 11. Assuming that it exists, a photographed image that can be obtained when the object 200 is photographed through the virtual electronic devices 100-1 and 100-2 is displayed on the display. That is, the side image of the object 200 that can be obtained when the object 200 is photographed at the B position 12 and the object 200 that can be obtained when the object 200 is photographed at the C position 13 ) is displayed together with the front image of the object 200 acquired in real time through the camera 110 of the electronic device 100. Through this, the user may simultaneously observe captured images of the object 200 from a plurality of viewpoints without using a plurality of electronic devices 100 .

Meanwhile, the photographed images of the object 200 at positions other than the above-described position of the electronic device (position B 12 and position C 13) are estimated to be obtained when the object is photographed. means To this end, the electronic device 100 generates a 3D image 210 of the object 200, and based on the 3D image 210, the electronic device at position B 12 and position C 13 ( When the object 200 is photographed by 100, images that can be obtained are respectively estimated. Hereinafter, with reference to FIGS. 2 to 14, an embodiment of the present disclosure related thereto will be described in detail.

2 is a schematic configuration diagram of an electronic device according to an embodiment of the present disclosure. 3 is a schematic flowchart of a method for controlling an electronic device according to an embodiment of the present disclosure. FIG. 4A is an exemplary view for explaining relatively identifying a second location based on a first location according to an embodiment of the present disclosure, and FIG. 4B is a view illustrating a first location and a second location according to another embodiment of the present disclosure. It is an exemplary diagram for explaining location identification.

Referring to FIG. 2 , an electronic device 100 according to an embodiment of the present disclosure includes a plurality of cameras 110, a display 120, a memory 130, and a processor 140.

According to an embodiment of the present disclosure, the electronic device 100 includes a plurality of cameras (a first camera 111, a second camera 112, and a third camera 113), but through the camera 110. It refers to an electronic device including a display 120 capable of displaying an acquired captured image. According to an embodiment of the present disclosure, the electronic device 100 may be implemented as a smart phone, a tablet PC, a digital camera, a laptop computer, or the like.

Meanwhile, the electronic device 100 includes a plurality of cameras ((the first camera 111, the second camera 112, and the third camera 113). The plurality of cameras 110 are As a configuration for obtaining a photographed image of the electronic device 100, it may be provided on the front or rear of the electronic device 100, and each of the plurality of cameras 111, 112, and 113 may include an image sensor and a lens.

Here, the field of view (FOV) of the lenses included in each of the cameras 111, 112, and 113 may be different from each other. For example, the electronic device 100 includes a plurality of cameras each including a telephoto lens, a wide angle lens, and a super wide angle lens having different angles of view disposed on the rear surface. (111, 112, 113). Meanwhile, in FIG. 2 , it is illustrated as including first to third cameras, but there is no particular limitation on the number and type of cameras according to the present disclosure.

Meanwhile, when a control application (eg, a photographing application) of the electronic device 100 is executed or a control input related to photographing is input on the executed application, the camera 110 included in the electronic device 100 is driven. It can be. For example, upon receiving a photographing command from a user, the processor 140 may control the camera 110 to execute photographing and obtain an image of a specific object. Here, the captured image includes both moving images and still images.

The display 120 may display various screens. For example, the display 120 may display an image captured through the camera 110 or display a user interface (UI) for controlling a photographing processor through the camera 110 . In addition, the display 120 may display images obtained in real time through the camera 110 as well as images corresponding to other viewpoints of objects included in the images. This will be described later in detail.

The display 120 may include various types of display panels such as a liquid crystal display (LCD) panel, an organic light emitting diodes (OLED) panel, a plasma display panel (PDP) panel, an inorganic LED panel, and a micro LED panel. It is not limited. Meanwhile, the display 120 may constitute a touch screen together with a touch panel.

The memory 130 may store information supporting various functions of the electronic device 100 . For example, the memory 130 may store view angle information of a plurality of lenses included in the camera 110 . Also, the memory 130 may store a driving program of the electronic device 100 and a plurality of applications, or data and commands used in them. At least some of the applications may be downloaded from an external server (not shown) through wireless communication. Also, at least some of the applications may exist for basic functions according to the characteristics of the electronic device 100 . Meanwhile, the application may be stored in the memory 130 and driven by the processor 140 to perform an operation (or function) of the electronic device 100 .

The processor 140 controls overall operations of the electronic device 100 . To this end, the processor 140 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). .

Referring to FIG. 3 , according to an embodiment of the present disclosure, the processor 140 identifies one camera among a plurality of cameras based on a first location of the electronic device and a second location set based on the first location ( S310).

Here, the first position means a position when the processor 140 captures an object using at least one camera among the plurality of cameras 111 , 112 , and 113 in real time. That is, the first location may correspond to the location of the electronic device 100 .

Meanwhile, the second position is a position other than the first position corresponding to the real-time position of the electronic device 100 acquiring the image of the object, and means the position of the virtual electronic device. Specifically, the processor 140 may not only obtain a captured image of the object using the camera 110, but also obtain an image of the object from another viewpoint through a virtual camera. In this case, the virtual camera The position of may correspond to the second position. For example, referring to FIG. 1 , positions B 12 and C positions 13, which are positions of virtual electronic devices 100-1 and 100-2 other than the electronic device 100, are the second positions. may apply to

On the other hand, in FIG. 1, it is assumed that the second position is set to two places for understanding of the present disclosure, but the skilled person can clearly understand that the number of second positions can be set to more or less. will be.

According to an embodiment of the present disclosure, the processor 140 identifies the first location. Also, the processor 140 identifies a second location set based on the first location. Referring to FIG. 1 , the processor 140 identifies the real-time location of the electronic device 100 and sets second locations 12 and 13 where virtual electronic devices 100-1 and 100-2 are identified as existing. can identify. For example, the processor 140 identifies a coordinate value corresponding to a first position on a WGS84 coordinate system or a longitude and latitude coordinate system based on a GPS sensor (not shown) included in the electronic device 100, and Based on the coordinate value corresponding to the location, a second location set to correspond to the first location or a coordinate value corresponding to the second location may be identified.

More specifically, the processor 140 may identify a second location set relatively to the first location based on the first location, that is, the location of the electronic device 100 . To this end, the processor 140 first identifies the current location of the electronic device 100 . Also, the processor 140 may identify the second position based on a preset displacement angle and a preset displacement distance based on the first position. Referring to FIG. 4A , as described above, the second position may be set relative to the first position. The first position corresponds to the position of the camera 110 capturing an object, that is, the position of the electronic device 100 . In this case, the second position may be set to a position of a preset angle and distance based on the first position. Referring to FIG. 4A, the second position is 1) an angle (a°) in the counterclockwise horizontal direction on the xy plane and 2) an angle (b°) in the vertical direction of the upper direction of the z axis. It can be set to be located at . At this time, when the first position, that is, the position of the electronic device 100 is changed from A(16) to A'(16-1), the second position is also changed from B(18) to B'(18-1). . Accordingly, when the processor 140 detects a change in the first location as the location of the electronic device 100 changes, the processor 140 regenerates the changed coordinate values of the first location and the changed coordinate values of the second location according to the first location change. need to identify

Meanwhile, according to another embodiment of the present disclosure, referring to FIG. 4B , the processor 140 identifies a second location located at a preset angle and distance from the first location, and then changes the first location and Regardless, the second location can be identified at a fixed location. Referring to FIG. 4B , the second position is located at 1) an angle (a°) in a counterclockwise horizontal direction and 2) an angle (b°) in a vertical direction in an upper direction from the first position. At this time, even if the first position, that is, the position of the electronic device 100 is changed from A(16) to A′(16-1), the processor 140 may still identify the second position as B(18). . In this case, after identifying the second location, the processor 140 does not identify the coordinate values of the second location again even if the first location changes due to a change in the location of the electronic device 100 .

Meanwhile, according to an embodiment of the present disclosure, the processor 140 may set a reference coordinate system based on the object 200 detected through the camera 110 in order to identify the first location and the second location. . More specifically, distance information (or depth information) to an object is acquired using a TOF (Time of Flying) sensor (not shown) included in the electronic device, and the position of the object is determined based on the obtained distance information. A reference coordinate system can be set as In this case, the position of the object may correspond to the origin of the reference coordinate system. And, within the set reference coordinate system, the processor 140 identifies a coordinate value corresponding to the first position, that is, the position of the electronic device 100, and coordinates corresponding to the second position set based on the calculated coordinate value. value can be identified. That is, the processor 140 may obtain the coordinate information of the electronic device 100 on the WGS84 coordinate system or the longitude and latitude coordinate system as location information about the first location by using a GPS module (not shown), or the processor ( 140 may also obtain location information about the first location of the coordinate information of the electronic device 100 on the reference coordinate system set with respect to the object photographed through the camera 110 . However, it is not limited thereto.

Referring back to FIG. 3 , according to an embodiment of the present disclosure, the processor 140 identifies a first location and a second location set based on the first location, and then selects among a plurality of cameras based on the second location. One camera is identified (S320). That is, the processor 140 may select one camera used to photograph an object from among a plurality of cameras based on the identified second location.

To this end, according to an embodiment of the present disclosure, the processor 140 may use view angle information of each of a plurality of cameras. Here, the angle of view information means a photographing range of each lens included in each camera. The processor 140 may identify a camera corresponding to an angle of view including the second position among the plurality of cameras as one camera. As described above, the angles of view of the plurality of cameras included in the electronic device 100 may be set differently. Accordingly, the processor 140 identifies the angle-of-view range of each camera for the object after identifying the second position. Then, it is identified whether the second position is included in the angle-of-view range of each camera. In addition, the processor 140 may identify a camera corresponding to the angle of view including the second position as one camera, and then acquire an image of the object using the identified camera.

Meanwhile, according to an embodiment of the present disclosure, the processor 140 may identify a camera corresponding to a view angle including the second position based on a reference coordinate system set for an object. Specifically, the processor 140 determines the second position (or a coordinate value in the reference coordinate system corresponding to the second position) within the reference coordinate system set for the object and the angle-of-view range of each camera 110 for the object. identify In addition, the processor 140 may identify a field of view range including the second position, and identify a camera corresponding to the identified field of view range or field of view as one camera.

Referring back to FIG. 3 , after identifying one camera, the processor 140 acquires a first object image by photographing an object through the identified camera (S330). Specifically, the processor 140 receives a user input for operating the camera 110 through the display 120 or the interface, or the processor 140 receives a predetermined condition (eg, a preset condition for the object) for acquiring the first object image. When the focal length and/or angle) is identified as being satisfied, a first object image is acquired by photographing the object while the electronic device is in the first position.

Here, the first object image refers to an image obtained in real time after the processor 140 captures an object using the identified camera 110 . Meanwhile, the first object image 20 may include a plurality of images. For example, when the processor 140 captures a video of the object 200 using the camera 110, the first object image 20 may include a plurality of images corresponding to each frame. there is.

In this case, the first object image 20 acquired by the processor 140 may include location information about the first location. For example, referring to FIG. 1 , when the processor 140 captures an object using the camera 110 at location A 11 , the first location corresponds to location A 11 . In this case, the first object image 20 data obtained by the processor 140 using the camera 110 may include location information about the A location 11 as metadata.

Meanwhile, in this specification, the second object image 30 refers to an image of an object that can be obtained from a second location. Unlike the first object image obtained through the camera, the second object image 30 is an image of an object estimated to be obtained by a virtual electronic device in a second location. The second object image 30 may be created using the first object image according to an embodiment.

Referring back to FIG. 3 , according to an embodiment of the present disclosure, the processor 140 obtains a first object image, and then, based on the 3D image of the object and the second location, the processor 140 corresponds to the second location. A conversion function is identified (S330). And, based on the first object image and the identified conversion function, a second object image corresponding to the second location is acquired (S340).

First, according to an embodiment of the present disclosure, the processor 140 to identify a transform function corresponding to the second position. A 3D image of the object may be generated, and a transformation function corresponding to the second position may be identified based on the generated 3D image. Hereinafter, the present disclosure related to this will be described.

5 is a flowchart schematically illustrating a method of identifying a transform function corresponding to a second position based on a 3D image of an object, according to an embodiment of the present disclosure.

Referring to FIG. 5 , according to an embodiment of the present disclosure, the processor 140 identifies one of the plurality of cameras (S311), and then captures the object 200 from a plurality of viewpoints through the identified camera 110. By photographing, a plurality of object images corresponding to a plurality of viewpoints may be acquired (S312). Further, the processor 140 may obtain a 3D image 210 of the object 200 based on the object identified in each of the plurality of object images (S313).

Specifically, the processor 140 acquires a plurality of images of the object 200 at various locations. That is, the processor 140 may obtain a plurality of images as a plurality of scan data of the object in order to generate the 3D image 210 of the object 200 . At this time, the processor 140 inputs parameters such as the focal length of the camera 110 preset for the object and/or the distortion of the lens 111 included in the camera 110, and the object 200 from a plurality of viewpoints. You will be able to obtain an image of For example, the processor 140 may acquire a captured image of the object 200 when a condition of the same focal distance of the camera 110 to the object 200 or a focal distance within a preset error range is met.

Also, according to an embodiment of the present disclosure, the processor 140 may display a UI on the display to guide the user to capture a plurality of images of the object. For example, when an object is detected through the identified camera 110, the processor 140 sets a reference coordinate system set for the object, and sets a plurality of positions (eg, each other) within the set reference coordinate system. A UI for guiding photographing of an object in another direction) may be displayed on the display 110 . Further, the processor 140 terminates the UI displayed on the display 110 when it is identified that a predetermined number of images (more specifically, the number of images necessary for generating a 3D image) of the object have been obtained.

Meanwhile, the processor 140 obtains a plurality of object images corresponding to a plurality of viewpoints, and then obtains a 3D image 210 of the object 200 based on an object identified in each of the plurality of object images. It can (S313). Specifically, the processor 140 may generate the 3D image 210 of the object 200 from a plurality of images based on triangulation. At this time, the processor 140 performs a calibration process on a plurality of images corresponding to a plurality of viewpoints with respect to the object 200, and based on the result of the calibration process, a 3D image of the object 200 is obtained. information can be obtained. To this end, programs such as MeshLab, netfabb, and MeshMixer may be stored in the memory 130 to generate a 3D image 210 of the object 200 using a plurality of images. The processor 140 may generate a 3D image 210 of the object 200 using a stored 3D image generating program.

Meanwhile, the processor 140 may use a deep learning-based object detection model (hereinafter, a deep learning-based first model) to identify an object through a camera. The deep learning-based object detection model may include a RCNN model, an FRCNN model, or a YOLO model. For example, the processor 140 inputs at least one object image among a plurality of object images to any one of the RCNN model, the FRCNN model, or the YOLO model stored in the memory 130, thereby generating an object included in each object image. can detect

Meanwhile, in FIG. 5 , it is shown that the processor 140 obtains a 3D image of the object prior to acquiring the first object image. However, according to an embodiment of the present disclosure, the processor 140 may acquire a 3D image of an object based on a plurality of images including the first object image after acquiring the first object image. will be.

Also, as another embodiment of the present disclosure, the processor 140 may obtain a 3D image of the object by inputting the first object image to a 3D image generation model based on deep learning. This will be described later in detail.

The processor 140 may obtain a 3D image 210 of the object 200 by inputting the first object image 20 to a neural network model (hereinafter, a second model based on deep learning). In this case, the neural network model includes a deep learning-based 3D image generation model. The deep learning-based 3D image generation model, when at least one 2D image of a specific object is input, estimates a 3D image 210 for a specific object based on the input at least one 2D image. It is a neural network model that has been trained to generate At this time, a generation model of a 3D image based on deep learning may be pre-stored in the memory 130 .

The first model based on deep learning used to acquire the 3D image 210 may be pre-learned so that when the 2D image of each object is input, the 3D image 210 of the object is output. According to an embodiment of the present disclosure, the neural network model may be a deep learning-based generative adversarial network (GAN) model. At this time, the GAN model is learned based on the 2D image and the 3D image (actual data) of each object. Specifically, when the processor 140 inputs a 2D image of a specific object, a 3D image 210 of the corresponding object is generated based on the 2D image by a generator, and a discriminator ) determines the 3D image 210 corresponding to the actual data of the object and the 3D image 210 generated by the generative model. At this time, if the 3D image of the actual data of the discrimination model and the 3D image 210 generated by the generation model are distinguished, the synaptic weight of the generation model is corrected. On the other hand, for a specific object, if the discrimination model fails to discriminate between a 3D image corresponding to actual data and the 3D image 210 generated by the generative model, learning of the object is terminated.

In this way, based on the model for which the three-dimensional image 210 generation of the plurality of objects has been pre-learned, the processor 140 converts the first object image 20 into only the first object image 20 that is a two-dimensional image. A 3D image 210 of the object 200 included in may be generated.

Meanwhile, according to an embodiment of the present disclosure, in order to acquire a plurality of images, the processor 140 detects the object 200 through the camera 110 and performs tracking of the object 200. You may. For example, the processor 140 uses a lidar sensor (not shown) included in the electronic device 100 to perform cloud data about the object 200 included in a plurality of images acquired through a camera. generate Also, the processor 140 may set a region of interest (ROI) including the object 200 in the cloud data. Thereafter, the processor 140 registers the 3D image 210 of the object 200 and the cloud data so that the object 200 is continuously displayed even if the location of the electronic device 100 or the object 200 is changed. can track

Hereinafter, with reference to FIGS. 6 to 8 , an embodiment of the present disclosure for identifying a transform function corresponding to a second position based on a 3D image and a second position will be described in detail.

As described above, according to an embodiment of the present disclosure, the processor 140 identifies a transform function corresponding to the second position based on the 3D image and the second position. Here, the transformation function means a function that transforms the first object image 20 obtained at the first location into a second object image, and the transformation function may include a transformation matrix.

According to an embodiment of the present disclosure, the processor 140 calculates a displacement angle of the second position based on the first position, which is the position of the electronic device, and identifies a conversion function corresponding to the calculated displacement angle. Alternatively, the processor 140 calculates a displacement distance of the second position based on the first position, which is the position of the electronic device, and identifies a conversion function corresponding to the calculated displacement distance. For example, the processor 140 determines a correlation between a first location (eg, coordinate values of the first location) and a second location (coordinate values of the second location) based on the 3D image 210. and a conversion function for converting the first object image into the second object image 30 corresponding to the second location based on the identified correlation.

6 is a schematic diagram of a control method of an electronic device for acquiring a transformation matrix based on coordinate values of feature points of an object and obtaining a second object image using the transformation matrix, according to an embodiment of the present disclosure; is a flow chart.

Meanwhile, according to an embodiment of the present disclosure, the processor 140 may identify a transform function based on a 3D image of an object. Specifically, referring to FIG. 6 , the processor 140 identifies feature points of the object identified at the first location based on the 3D image 210, and corresponds to the feature point identified at the first location. A first coordinate value is obtained (S331). Then, the processor 140 identifies a feature point of the object identified at the second location based on the 3D image, and obtains a second coordinate value corresponding to the feature point identified at the second location (S332), Based on the first coordinate value and the second coordinate value, a transformation matrix corresponding to the second position may be identified (S333).

First, the processor 140 obtains a coordinate value corresponding to a characteristic point of an object identified at each of a plurality of viewpoints based on the 3D image 210 of the object. For example, through the Harris Corner Detect method, the processor may acquire coordinate values for each feature point after detecting feature points of an object identified at each of a plurality of viewpoints. Alternatively, feature points of the object 200 may be detected using a lidar sensor (not shown) included in the electronic device 100 . In this case, the coordinate values may be obtained on a reference coordinate system, a WGS84 coordinate system, or a longitude and latitude coordinate system set by the processor 140 described above.

Meanwhile, in order to obtain coordinate values corresponding to feature points of the object, the processor 140 may use a plurality of object images corresponding to a plurality of viewpoints used to obtain the 3D image 210 . In this case, the plurality of viewpoints may include the first location and/or the second location. That is, in order to obtain a 3D image 210 of the object, the processor 140 photographs the object 200 from a plurality of viewpoints including the first position and/or the second position through the camera 110. In this case, information on feature points of the object 200 may be included in the plurality of object images corresponding to the plurality of viewpoints. Also, the processor 140 may obtain coordinate values corresponding to feature points in each object image.

The processor 140 acquires coordinate values corresponding to feature points of the object identified at each of a plurality of viewpoints (S351), and then obtains a plurality of transformation matrices for transformation between the obtained plurality of coordinate values (S352). . At this time, a plurality of transformation matrices are generated corresponding to the second position.

7 is an exemplary view illustrating obtaining a transformation matrix for transformation using coordinate values corresponding to feature points of an object according to an embodiment of the present disclosure.

Referring to FIG. 7 , the processor 140 detects eight feature points a1 to a8 of the object 200 included in the image acquired at the first location. Also, the processor 140 detects eight feature points b1 to b8 of the object 200 included in the image acquired at the second location. The processor 140 uses the coordinate values of the feature points of a1 to a8 and the coordinate values of the feature points of b1 to b8 to create a transformation matrix T1 for converting the image at the first position to the image at the second position. can be obtained. Specifically, based on the coordinate values of the feature points of a1 to a8 and the coordinate values of the feature points of b1 to b8 corresponding to the feature points of a1 to a8, the image at the first location is converted to the image at the second location. It will be possible to calculate a transform matrix that changes. Meanwhile, in FIG. 7, it has been described that a transformation matrix for one second position is acquired, but when a plurality of second positions are set, each transformation matrix corresponding to each second position will be obtained.

8 is an exemplary view illustrating obtaining an object image corresponding to a second position based on a transformation matrix according to an embodiment of the present disclosure.

Referring back to FIG. 3 , according to an embodiment of the present disclosure, the processor 140 converts a first object image into a second object image based on the identified conversion function, and the second object corresponding to the second position. An image 30 is obtained. Specifically, the processor 140 may generate the second object image 30 by transforming pixel values of the first object image based on the conversion function. Referring to FIG. 8 , the processor 140 applies a transformation matrix T1 to the first object image 20 in which the front, right side, and top side of the object 200 are observed, so that the right side of the object 200 is An enlarged second object image 30 may be obtained.

However, it is not limited thereto, and the processor 140 may generate the second object image 30 from the 3D image 210 . That is, an image obtainable when photographing the object 200 at the second location may be obtained from a pre-created 3D image 210 of the object 200 . Specifically, the processor 140 may acquire a second object image corresponding to the second location by capturing the 3D image of the object 200 at the second location. To this end, the processor 140 may perform a process of matching the 3D image 210 of the object 200 with the object in the first object image 20 recognized through the camera 110 . For example, the processor 140 may synchronize the reference coordinate system for the object and the reference coordinate system for the 3D image of the object. A second object image corresponding to the second position of the object may be obtained by capturing a 3D image of the object at a point corresponding to the second position in the reference coordinate system.

9 is an exemplary view illustrating obtaining a second object image 30 using a 3D image of an object according to an embodiment of the present disclosure.

Referring to FIG. 9 , it is assumed that the first position is A and the second position is B. At this time, the second object image 30 means an image that can be obtained when the processor 140 captures the 3D image 210 of the object in the reference coordinate system through the camera 110 at B. That is, the second object image 30 may be an image obtained by capturing a portion of the 3D image 210 of the object 200 .

To this end, as described above, the processor 140 may perform a process of matching the object 200 and the 3D image 210 of the object on the reference coordinate system. However, it is not limited thereto, and without a matching process, the processor 240 matches the reference coordinate system applied to the object 200 and the 3D image 210 of the object, thereby forming the 3D image 210 of the object. A second object image 30 corresponding to the second position of the object may be acquired. That is, unlike acquiring the second object image 30 using a conversion function, the processor 140 acquires the first object image 30 using the 3D image 210 when acquiring the second object image 30 . may not be used.

In addition, according to an embodiment of the present disclosure, the processor 140 obtains a second object image (hereinafter referred to as a 2-1 object image) obtained by applying a transform function to the first object image and a 3D image. The second object image corresponding to the second position may be finally obtained by synthesizing the second object image (hereinafter, 2-2 object image).

Meanwhile, according to an embodiment of the present disclosure, the second location may be changed by a user's input. Specifically, when the user changes the second position through the display 120 or the interface, the processor 140 may identify the second position and then obtain the second object image 30 corresponding to the changed second position. there is.

To this end, according to an embodiment of the present disclosure, the processor 140 calculates the first position and the angular change value and the displacement change value of the second position with respect to the first position, and the calculated angle change value and displacement change value Based on this, the transformation matrix can be updated. For example, after identifying the second position based on the first position, based on the second position, an angle change value in a 3D space is calculated, and a rotation transformation matrix is obtained based on the calculated angle change value. can

Meanwhile, after acquiring the transformation matrix, the processor 140 acquires the characteristic points of the object identified in the first object image 20 and the characteristic points of the object identified in the first object image 20 among the obtained plurality of transformation matrices. A second object image corresponding to the second position is obtained based on the transformation matrix corresponding to (S331_c). The transformation matrix corresponding to the feature point of the object identified in the first object image means a matrix for transforming the image corresponding to the first position described above into an image corresponding to the second position.

Meanwhile, according to an embodiment of the present disclosure, the processor 140 may differently apply a method of obtaining the second object image 30 according to the second location. Specifically, the processor 140 identifies view angle information of one identified camera based on the second position among the plurality of cameras. Further, the processor 140 identifies a reference range of the camera for the object based on the identified view angle information. And the processor 140 identifies the second position based on the reference range, and if the second position is included in the identified reference range, obtains the second object image 30 based on the first object image and the transformation matrix, , If the second position is not included in the identified reference range, a 3D image of the object acquired based on a plurality of object images or a 3D image obtained through a neural network model (deep learning-based 3D image generation model) Based on (210), the second object image 30 may be acquired.

Here, the reference range means a limit range of an object that can be acquired through a camera. In this case, the reference range may be set to correspond to the angle-of-view range of each lens included in the camera. For example, assuming that the angle of view of the lens is 120°, the reference range may be set to 120° for the electronic device based on the object.

10 is a diagram for explaining setting a reference range based on view angle information set in a camera and generating a second object image according to the reference range, according to an embodiment of the present disclosure.

Specifically, referring to FIG. 10 , the processor 140 identifies view angle information (eg, 120°) of one camera identified based on the second position, and determines a view angle for an object based on the identified view angle information. Assume that the reference range of the camera is identified as 120°. At this time, the processor 140 identifies the second position 12 located at B as being included in the identified reference range. However, the processor 140 identifies that the second position 13 located at C is not included in the reference range. In this case, the processor 140 generates the second object image 30 corresponding to the second position located at B by applying a transformation matrix to the first object image, and generates the second object image corresponding to the second position located at C. In the case of the image 30, it is created using the 3D image 210.

11 is an exemplary view of displaying a first object image and a second object image 30 on a display according to an embodiment of the present disclosure.

Referring back to FIG. 3 , after acquiring the second object image 30, the processor 140 displays the first object image 20 and the second object image 30 on the display 120 (S350). .

Referring to FIG. 11 , the display 120 not only displays the first object image 20 acquired through the camera 110 in real time, but also simultaneously displays the second object image 30 corresponding to the second position. do. Through this, the user can be provided with images taken from a plurality of viewpoints of a specific object using one camera 110, and quickly determine the optimal shooting viewpoint, angle, position, etc. for photographing the object. can decide

Meanwhile, since the second object image 30 is an image estimated based on the first object image 20 and the 3D image 210 of the object, it is different from an image obtained by photographing the object in the actual second position. there may be In this case, a process of correcting the second object image 30 provided or scheduled to be provided to the user is required. Hereinafter, specific embodiments of the present disclosure related to this will be described.

12 is a flowchart schematically illustrating a control method of an electronic device for updating a 3D image of an object based on an image of the object acquired at a second location according to an embodiment of the present disclosure. 13A and 13B are exemplary diagrams illustrating displaying a second object image 30 based on an updated 3D image.

Referring to FIG. 12 , the processor 140 acquires a third object image by photographing an object at a location different from the first location through the camera 110 (S360), and is included in the first object image 20. It is identified whether at least a part of the object included in the second object image 30 is included in the third object image (S30), and if at least a part of the object 200 is included in the third object image, When at least a part of the object included in the 3-object image is different from the second object image 30, the 3D image is updated based on the third object image (S380).

First, the processor 140 captures the object 200 at a location different from the first location through the camera 110 to obtain a third object image 40 (S360). At this time, the other location may include the second location. The third object image is common to those acquired through the camera 110 like the first object image, but there is a difference in each acquired position. Specifically, the third object image is an image obtained by the processor 140 through the camera 110 after acquiring the first object image.

The processor 140 identifies whether at least a part of the object 200 included in the second object image 30, which is not included in the first object image 20, is included in the third object image 40. (S370).

First, referring to FIG. 13A, the first position is located on the x-axis at t1, moves 45° counterclockwise on the xy plane from the x-axis at t2, and then is located on the x-axis at t3. At this time, the second position relatively set with respect to the first position was located on the y-axis at 90° in the counterclockwise direction of the x-axis on the xy plane at t1. Meanwhile, since the second position is set relative to the first position, the second position is also moved by the displacement angle of the first position at t2. Specifically, since the first position is moved 45° counterclockwise on the xy plane from the x axis, the second position is located at a point moved 45° counterclockwise on the xy plane from the y axis. And at t3 the second position is on the y-axis. Hereinafter, a first object image, a second object image, and a 3D image at the first location and the second location according to the location of the electronic device 100 in FIG. 13A will be described with reference to FIG. 143 . do.

Referring to FIG. 13B , the right side of the object 200 in the third object image 40 obtained by the processor 140 at time t2 includes a character. However, at the time point t1, the right side of the object 200 in the second object image 30 displayed by the processor 140 does not include a character. As described above, the processor 140 generates a first object image 20 in which a character cannot be identified on the right side of the object 200 and a 3D image 210 obtained based on the first object image 20 This is because the second object image 30 is obtained based on . Therefore, the processor 140 may, based on the acquired third object image 40, at least a part of the object 200 included in the second object image 30, although not included in the first object image 20. , for example, to identify whether it contains characters, shapes, shapes, etc. In other words, based on the third object image 40 , characteristic information of an object 200 that is not identified through the first object image 20 is identified.

Further, the processor 140 determines that at least a part of the object 200 is included in the third object image 40, and at least a part of the object included in the third object image 40 is different from the second object image 30. In this case, the 3D image 210 is updated based on the third object image 40.

Referring back to FIG. 13B , if a character included in the right side of the object 200 is identified based on the third object image 40 acquired by the processor 140 at time t2, then the first displayed at time t3 The second object image 30 will be updated to include a character on the right side of the object 200 in the second object image 30 . Referring to FIG. 13B , the second object images at time points t1 and t3 are images generated by assuming that they will be acquired at the same location. However, in the case of the second object image at time t3, a 3D image updated based on the third object image acquired at time t2 is generated. Therefore, the second object image at time t1 is different from the second object image at time t3, even though the images are assumed to be acquired at the same location.

Meanwhile, this updating process is not performed on the second object image 30 to be displayed later. For example, when the processor 140 stores the first object image 20 and the second object image 30 displayed on the display 120 in the memory 130, the previously stored second object image 30 ) may be updated based on the updated 3D image 210. For example, when the processor 140 captures and records a video of the object 200 through a camera and updates the 3D image 210 based on the third object image 40, at time t1 The second object image 30 may be updated to include a character on the right side of the object 200 in the second object image 30 stored in the memory.

Meanwhile, according to an embodiment of the present disclosure, the processor 140 acquires the second object image 30 corresponding to the second position based on the updated 3D image 210, and the second object image 30 ) can be displayed on the display 120.

Through this, the user may be provided with more accurate images of the object from a plurality of viewpoints.

14 is a detailed configuration diagram of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 14 , an electronic device 100 according to an embodiment of the present disclosure includes a camera 110, a display 120, a memory 130, a communication unit 150, a lidar sensor 160, and a GPS sensor 170. ), the TOF sensor 180, and the processor 140. Hereinafter, parts overlapping with the above description will be omitted or abbreviated.

The processor 140 may transmit and receive various types of information between the electronic device 100 and an external device through the communication unit 150 . For example, information about the neural network model of the object 200 may be received from an external device (eg, a server providing an electronic device control program), and other deep learning-based object detection models, such as electronic Information on various programs for controlling the device 100 may be received. The processor 140 may know various types of communication networks through the communication unit 150, for example, WLAN (Wireless LAN), Wi-Fi (Wi-Fi), LTE (Long Term Evolution), 5G, etc. A wireless communication method or a wired communication method such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), FTTH (Fiber To The Home) may be used.

Meanwhile, the processor 140 may obtain cloud data about the object 200 through the lidar sensor 160 . The processor 140 may obtain cloud data about the object 200 through the lidar sensor 160 and generate a 3D image 210 of the object 20 .

Also, the processor 140 may detect the first position of the electronic device 100 on the WGS84 coordinate system or the latitude and longitude coordinate system through the GPS sensor 170 . Specifically, the processor 140 may identify the first location based on the coordinate values of the electronic device.

In addition, the processor 140 obtains a plurality of images from a plurality of viewpoints, including the first object image 20 of the object using the camera 110 through the TOF sensor 180, and simultaneously acquires a plurality of images of the first object. Depth information may also be acquired. Based on such depth information, a 3D image 210 of the object 200 may be obtained based on a plurality of images.

Meanwhile, in the above description, steps S310 to S360 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed.

Meanwhile, according to an exemplary embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium (eg, a computer). A device is a device capable of calling a command stored from a storage medium and operating according to the called command, and may include a device according to the disclosed embodiments. When a command is executed by a processor, the processor directly or A function corresponding to a command may be performed using other components under the control of the processor. A command may include code generated or executed by a compiler or an interpreter. A device-readable storage medium is a non-transitory It can be provided in the form of a (non-transitory) storage medium, where 'non-transitory storage medium' only means that it is a tangible device and does not contain a signal (e.g. electromagnetic wave), This term does not distinguish between a case where data is stored semi-permanently and a case where data is temporarily stored in a storage medium, for example, 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored on a device-readable storage medium such as a memory of a manufacturer's server, an application store server, or a relay server. It can be temporarily stored or created temporarily.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is commonly used in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present disclosure.

100: electronic device
110: camera
120: display
130: memory
140: processor
200: object

Claims (18)

In electronic devices,
multiple cameras;
display;
Memory; and
Identifying one of the plurality of cameras based on a first location of the electronic device and a second location set based on the first location, and detecting the identified camera while the electronic device is in the first location. Obtaining a first object image by photographing an object, identifying a transformation function corresponding to the second location based on the 3D image of the object and the second location, and identifying the first object image and the transformation function. An electronic device comprising: a processor that obtains a second object image corresponding to the second position based on a function and displays the first object image and the second object image on the display. According to claim 1,
the processor,
Identifying angles of view of each of the plurality of cameras, and identifying a camera corresponding to an angle of view including the second position among the plurality of cameras as the one camera. According to claim 1,
the processor,
A plurality of object images corresponding to the plurality of viewpoints are obtained by photographing the object from a plurality of viewpoints through the camera, and a 3D image of the object is obtained based on an object identified in each of the plurality of object images. An electronic device for obtaining. According to claim 3,
the processor,
Based on the 3D image, a feature point of the object identified at a first location is identified, a first coordinate value corresponding to the feature point is obtained, and an object identified at a second location based on the 3D image An electronic device that identifies a feature point of , obtains a second coordinate value corresponding to the feature point, and identifies a transformation matrix corresponding to the second position based on the first coordinate value and the second coordinate value. . According to claim 1,
The memory includes a neural network model trained to generate a 3D image of an object based on an input object image,
the processor,
A 3D image of the object is obtained by inputting the first object image to the neural network model, and a second object image corresponding to the second position is obtained based on the 3D image obtained through the neural network model. electronic devices to acquire. According to claim 5,
the processor,
A reference range for the object is identified, and if the second position is included in the identified reference range, the second object image is acquired based on the first object image and a transformation matrix, and the second position is An electronic device that obtains the second object image based on the 3D image when it is not included in the identified reference range. According to claim 6,
the processor,
An electronic device that identifies the reference range based on the identified information on the angle of view of the camera. According to claim 5,
the processor,
When a third object image is obtained by photographing the object at a location different from the first location through the camera, at least a part of an object included in the second object image but not included in the first object image is displayed in the first object image. 3 identify whether the object is included in the image,
When at least a part of the object is included in the third object image and at least a part of the object included in the third object image is different from the second object image, the 3D image is based on the third object image. Electronic device to update. According to claim 7,
the processor,
An electronic device that obtains a second object image corresponding to the second location based on the updated 3D image and displays the second object image on the display. A method for controlling an electronic device including a plurality of cameras,
identifying one of the plurality of cameras based on a first location of the electronic device and a second location set based on the first location;
obtaining a first object image by photographing an object through the identified camera while the electronic device is in the first location;
identifying a transform function corresponding to the second position based on the 3D image of the object and the second position;
obtaining a second object image corresponding to the second position based on the first object image and the conversion function; and
and displaying the first object image and the second object image on a display. According to claim 10,
The step of identifying the one camera,
identifying angles of view of each of the plurality of cameras; and
and identifying, among the plurality of cameras, a camera corresponding to an angle of view including the second position as the one camera. According to claim 10,
acquiring a plurality of object images corresponding to the plurality of viewpoints by photographing the object from a plurality of viewpoints through the camera; and
And based on the object identified in each of the plurality of object images, obtaining a three-dimensional image of the object, the control method. According to claim 12,
Identifying the conversion function comprises:
based on the 3D image, identifying a feature point of the identified object at a first location, and obtaining a first coordinate value corresponding to the feature point;
based on the 3D image, identifying a feature point of the identified object at a second location, and obtaining a second coordinate value corresponding to the feature point; and
and identifying a transformation matrix corresponding to the second position based on the first coordinate value and the second coordinate value. According to claim 10,
Further comprising obtaining a 3D image of the object by inputting the first object image to a neural network model,
Obtaining the second object image,
Acquiring a second object image corresponding to the second position based on the 3D image obtained through the neural network model. According to claim 14,
Obtaining the second object image,
A reference range for the object is identified, and if the second position is included in the identified reference range, the second object image is acquired based on the first object image and a transformation matrix, and the second position is If not included in the identified reference range, obtaining the second object image based on the 3D image. According to claim 15,
The control method of claim 1 , wherein the reference range is identified based on the identified camera view angle information. According to claim 16,
obtaining a third object image by photographing the object at a location different from the first location through the camera;
identifying whether at least a part of an object included in the second object image but not included in the first object image is included in the third object image; and
When at least a part of the object is included in the third object image and at least a part of the object included in the third object image is different from the second object image, the 3D image is based on the third object image. Further comprising the step of updating the control method. According to claim 17,
In the step of displaying the second object image,
A control method of obtaining a second object image corresponding to the second position based on the updated 3D image, and displaying the second object image on the display.

Images