WO2021133053A1

WO2021133053A1 - Electronic device and method for controlling same

Info

Publication number: WO2021133053A1
Application number: PCT/KR2020/018980
Authority: WO
Inventors: 조승현; 송석우; 이요한
Original assignee: 삼성전자주식회사
Priority date: 2019-12-26
Filing date: 2020-12-23
Publication date: 2021-07-01
Also published as: KR20210083016A

Abstract

The present disclosure provides an electronic device and a method for controlling same. The electronic device of the present disclosure comprises a camera, a display, and a processor which: controls the display to display an image obtained through the camera and display an AR object on the image; when the depth of a point at which the AR object is located on the image is changed in a state where the electronic device is in a normal mode, adjusts the size of the AR object displayed on the display, on the basis of the depth that is changed; and when the depth of a point at which the AR object is located on the image is changed in a state where the electronic device is in a batch mode, performs control to maintain the size of the AR object displayed on the display.

Description

Electronic device and its control method

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device displaying an AR object and a control method thereof.

As electronic technology develops, technologies using augmented reality (AR) are becoming common among users.

AR is a technology that combines and displays a virtual object (or information) in an image taken of the physical space of the real environment. Through the virtual object displayed on the display, it may be displayed as if the virtual object exists together in the real space, and useful information related to the real object may be provided to the user.

On the other hand, the user may recognize the perspective (or distance) of the object by recognizing the small size of the object located far from the user in the real space and recognizing the large size of the object located close to the user.

In addition, since the AR space (ie, a virtual space grafted with reality) is composed of three dimensions like the real space, the size of the virtual object placed in the AR space varies according to the distance from the user (or electronic device). is changed

However, when the distance between the user and the virtual object in the AR space is too close, the size of the virtual object becomes very large, and when the distance between the user and the virtual object is too far, the size of the virtual object becomes very small, There was a problem that it was difficult to recognize a virtual object or to control (or manipulate) the virtual object.

The present disclosure has been made in response to the above-mentioned necessity, and an object of the present disclosure is to provide an electronic device for adjusting the size of an AR object and a method for controlling the same.

In order to achieve the above object, an electronic device according to an embodiment of the present disclosure controls the display to display an AR object on a camera, a display, and an image acquired through the camera, and the mode of the electronic device is a normal mode When the depth of the point at which the AR object is located on the image is changed in the in state, the size of the AR object displayed on the display is adjusted based on the changed depth, and in a state that the mode of the electronic device is the arrangement mode, and a processor that controls to maintain the size at which the AR object is displayed on the display when the depth of the point where the AR object is located on the image is changed.

Here, the depth of the point at which the AR object is located may be changed as the point at which the camera is located or the point at which the AR object is located on the image moves according to the depth direction.

On the other hand, the processor controls the display to display the AR object on the image and the image based on the size of the AR object, and in the arrangement mode, when the depth of the point where the AR object is located on the image is changed, the AR object is displayed on the display. You can change the size of the AR object so that it retains its displayed size.

Here, the processor increases the size of the AR object to maintain the size at which the AR object is displayed on the display when the depth of the point where the AR object is located increases while the mode of the electronic device is the arrangement mode, and the AR object is positioned When the depth of the point of interest decreases, the size of the AR object may be reduced to maintain the size at which the AR object is displayed on the display.

On the other hand, the processor controls the display to display the AR object on the image and the image based on the size of the AR object, and in a state where the mode of the electronic device is the normal mode, allocates the depth of the point where the AR object is located and the AR object Based on the specified properties, the size of the AR object displayed on the display can be adjusted.

Here, the processor controls the size at which the AR object is displayed on the display to be maintained when the property assigned to the AR object is a property in which the size is fixed while the mode of the electronic device is the normal mode, and the property assigned to the AR object When this size is a variable property, the size of the AR object displayed on the display may be adjusted based on the changed depth.

Here, the processor maintains the size at which the AR object is displayed on the display when the depth of the AR object on the image is changed when the property assigned to the AR object is a fixed size while the mode of the electronic device is the normal mode. In a state where the mode of the electronic device is in the normal mode, if the property assigned to the AR object is a property whose size changes, the AR is displayed on the display so that the size decreases when the depth of the point where the AR object is located on the image increases. The size at which the object is displayed may be adjusted, and the size at which the AR object is displayed on the display may be adjusted so that the size increases when the depth of the point where the AR object is located decreases.

A method of controlling an electronic device according to an embodiment of the present disclosure includes displaying an image acquired through a camera and an AR object on the image, and in a state in which the mode of the electronic device is a normal mode, the AR object is positioned on the image If the depth of the point is changed, adjusting the size at which the AR object is displayed based on the changed depth, and in a state that the mode of the electronic device is the arrangement mode, the depth of the point where the AR object is located on the image is changed and controlling the AR object to maintain the displayed size.

Meanwhile, in the displaying step, the AR object is displayed on the image and on the image based on the size of the AR object, and in the controlling step in the arrangement mode, when the depth of the point where the AR object is located on the image is changed, the AR object You can change the size of the AR object so that it retains its displayed size.

Here, the controlling includes increasing the size of the AR object to maintain the displayed size when the depth of the point where the AR object is located increases in the arrangement mode, and in the arrangement mode, the AR object The method may include reducing the size of the AR object to maintain the size at which the AR object is displayed when the depth of the point at which is located is decreased.

On the other hand, the displaying step displays the image and the AR object on the image based on the size of the AR object, and the adjusting step is in the normal mode, in the depth of the point where the AR object is located and the properties assigned to the AR object. Based on this, the size at which the AR object is displayed can be adjusted.

Here, the adjusting step is a step of controlling so that the displayed size of the AR object is maintained when the property assigned to the AR object is a property with a fixed size in the normal mode, and assigning the AR object to the AR object in the normal mode When the changed property is a property whose size is changed, the method may include adjusting the displayed size of the AR object to be changed based on the changed depth.

Here, the adjusting includes controlling the size at which the AR object is displayed to be maintained when the depth of the AR object is changed on the image when the property assigned to the AR object is a property in which the size is fixed in the normal mode; In the normal mode, if the property assigned to the AR object is a property that changes in size, adjust the displayed size of the AR object so that the size decreases if the depth of the point where the AR object is located on the image increases, If the depth of the point where is is decreased, the size of the AR object displayed can be adjusted so that the size increases.

According to various embodiments of the present disclosure as described above, it is possible to provide an electronic device for adjusting the size of an AR object and a method for controlling the same.

In addition, the visibility of the AR object and the operation convenience of the AR object may be improved.

1 is a diagram for describing an electronic device according to an embodiment of the present disclosure.

2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

3 is a block diagram illustrating an additional configuration of an electronic device according to an embodiment of the present disclosure.

4A is a diagram for explaining the size of an AR object according to an embodiment of the present disclosure.

4B is a diagram for explaining the size of an AR object according to an embodiment of the present disclosure.

5 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

6 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

7 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

8 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

9 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

10 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

11 is a diagram for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

12 is a view for explaining an arrangement mode according to an embodiment of the present disclosure.

13A is a diagram for describing an arrangement mode according to an embodiment of the present disclosure.

13B is a diagram for describing an arrangement mode according to an embodiment of the present disclosure.

13C is a diagram for describing an arrangement mode according to an embodiment of the present disclosure.

14 is a diagram for explaining a normal mode according to an embodiment of the present disclosure.

15A is a diagram for describing a normal mode according to an embodiment of the present disclosure.

15B is a diagram for explaining a normal mode according to an embodiment of the present disclosure.

16 is a diagram for describing a flowchart according to an embodiment of the present disclosure.

-

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, the following examples may be modified in various other forms, and the scope of the technical spirit of the present disclosure is not limited to the following examples. Rather, these embodiments are provided to more fully and complete the present disclosure, and to fully convey the technical spirit of the present disclosure to those skilled in the art.

It is to be understood that the techniques described in the present disclosure are not intended to be limited to specific embodiments, and include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, like reference numerals may be used for like components.

As used in the present disclosure, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the above components.

In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

In this disclosure, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It is to be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component) When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression "configured to (or configured to)" as used in this disclosure depends on the context, for example, "suitable for," "having the capacity to" ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a processor configured (or configured to perform) A, B, and C” refers to a dedicated processor (eg, an embedded processor) for performing the above operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing the above operations.

As shown in FIG. 1 , the electronic device 100 is a device capable of providing augmented reality (AR), and may be implemented as a smart phone or the like. Here, augmented reality may refer to combining a virtual object (hereinafter, augmented reality object; AR object) with an image based on a real environment (space or object).

The electronic device 100 may display an image and an AR object on the image. That is, the electronic device 100 may combine an image and an AR object, and display the AR object by superimposing it on the image.

Here, the image may include a photographed subject. Here, the subject refers to the real environment 10 located in the photographing area. In addition, the image can be recognized as a three-dimensional space through simultaneous localization and mapping (SLAM). Also, the image may be a plurality of image frames photographed in units of a preset frame rate.

Meanwhile, the AR object may be an image in which an existing object is rendered in 2D or 3D. For example, the AR object may be implemented in the same form as a 2D or 3D rendered image of objects such as a TV, a digital picture frame, a sound bar, a refrigerator, a washing machine, furniture, a car, a building, and a tree. However, this is only an embodiment, and the AR object is not limited thereto, and may be implemented in the form of various information such as text, text, image, photo, video, document, dashboard, and the like. Also, the AR object may be displayed in a translucent state.

Also, the AR object may be located at a specific point on an image recognized as a 3D space. In this case, a specific point on the image may be represented by three-dimensional space coordinates. For example, in three-dimensional space coordinates (x, y, z), the x-axis represents the horizontal direction, the y-axis represents the vertical direction, and the z-axis represents the depth direction. can

Also, a point on the image where the AR object is located may have a relative positional relationship with a point where a subject (eg, a floor, a plane, a wall, an object, etc.) included in the image is located. For example, if the point of view (PoV) at which the image was taken (or the point of view of the camera) is changed (or moved), the AR object in the same direction as the direction in which the point where the subject is located is changed (or moved) The point where is located may be changed (or moved). Here, the depth may indicate a distance between a point where the electronic device 100 (or camera) is positioned and a point where the AR object is positioned in a three-dimensional space on an image along the z-axis (depth direction).

Meanwhile, for the AR object, the size (default size) of the AR object may be preset for each AR object. In this case, the size of the AR object may indicate a size displayed in the standard depth, and the size may collectively refer to various concepts such as area, volume, diagonal length, horizontal and vertical length, radius, and diameter. Here, the standard depth may indicate a distance (eg, a distance on a z-axis) that is a reference so that the AR object is displayed in a preset size. For example, the standard depth may be preset to a distance of 1 m, which is an example and may be changed to various distances. Meanwhile, the size of the AR object may be changed according to a user command.

When the depth of the point where the AR object located on the image is located is a preset standard depth, the electronic device 100 may display the AR object in a preset size. For example, assuming that the size of an AR object, such as a 100-inch TV, is preset to be 220 x 125 in height, the electronic device 100 sets the depth of the AR object (100-inch TV) located on the image to 1 m, which is the standard depth. When , the AR object may be displayed in a preset size of 220 x 125 in height.

When the depth of the point where the AR object is located on the image is changed, the electronic device 100 according to an embodiment of the present disclosure may adjust or maintain the displayed size of the AR object according to the mode of the electronic device.

Here, the mode may be one of a normal mode and a batch mode. For example, the normal mode is a mode in which the size at which the AR object is displayed may be changed according to the depth of the point where the AR object is disposed (the distance between the electronic device (or camera) and the AR object). In addition, the arrangement mode is a mode in which a point where an AR object is located in an image can be changed according to a user command, and the size at which the AR object is displayed can be maintained regardless of the depth of the point where the AR object is placed. That is, the normal mode is a mode to which the perspective for the AR object is applied, and the arrangement mode is a mode to which the perspective to the AR object is not applied.

According to various embodiments of the present disclosure as described above, it is possible to provide an electronic device for adjusting the size of an AR object and a method for controlling the same. In addition, the visibility of the AR object and the operation convenience of the AR object may be improved.

Meanwhile, although the electronic device 100 is illustrated as a smartphone in FIG. 1 , this is only an example, and the electronic device 100 may be implemented as a wearable device that a user can wear. Here, the wearable device may be an accessory type (eg, watch, ring, bracelet, anklet, necklace, eyeglasses, contact lens, or head-mounted-device (HMD)), a textile or an integrated garment (eg, electronic garment). ), body-attached, or bioimplantable circuit, etc. In addition, the electronic device 100 may include a tablet PC, a speaker, a mobile phone, a telephone, an e-book reader, a desktop PC, a laptop PC, etc. , workstation, server, PDA, PMP (portable multimedia player), MP3 player, medical device, camera, television, DVD (digital video disk) player, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, set-top box , home automation control panel, security control panel, media box (eg Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game console (eg XboxTM, PlayStationTM), electronic dictionary, electronic key, camcorder, electronic picture frame or wearable It may be implemented as a device, etc. Also, the electronic device 100 may be implemented as a device having a transparent display or a flexible display in some cases.

In addition, the term augmented reality (AR) described in the present disclosure is not limited to the field of augmented reality, virtual reality (VR), mixed reality (MR), extended reality (eXtended Reality; XR), etc. can also be applied.

Hereinafter, an electronic device providing augmented reality and a control method thereof will be described in more detail with reference to the drawings.

2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure, and FIG. 3 is a block diagram illustrating an additional configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 according to an embodiment of the present disclosure may include a camera 110 , a display 120 , and a processor 130 .

The camera 110 may acquire an image. Specifically, the camera 110 may acquire an image by photographing a subject existing within a field of view (FoV) of the camera 110 at a specific point of view (PoV) of the camera 110 . Also, the camera 110 may sequentially acquire a plurality of images through continuous shooting.

The image acquired through the camera 110 or metadata separate from the image may include information about a frame rate, time, viewpoint, angle of view, etc. captured by the camera 110 . Here, the frame rate represents the number of frames (the number of images) acquired per second (or per minute), and the angle of view represents the focal length of the lens of the camera 110 and that of the camera 110 . A value determined according to a size (eg, a diagonal length) of an image sensor (not shown) may be represented. Also, the viewpoint may be detected by a sensor (eg, a gyro sensor, an acceleration sensor, etc.) provided inside or outside the camera 110 .

To this end, the camera 110 according to an embodiment of the present disclosure may be implemented as an RGB camera or a stereo camera.

The RGB camera may include a lens (not shown), an image sensor (not shown), and an image processor (not shown). The lens collects or diverges the light reflected from the object to the image sensor, and the image sensor divides the transmitted light into pixels and detects R (Red), G (Green), B (Blue) colors for each pixel. The signal is generated, and the image processor processes each pixel according to the electrical signal detected by the image sensor to obtain an image representing the color, shape, contrast, etc. of the object.

Here, the image is a projection of a real three-dimensional space onto a virtual two-dimensional plane, and each point (pixel) on the two-dimensional plane constituting the image contains two-dimensional position information (eg, position on the x-axis, position on the y-axis). may include In this case, the image processor (or the processor 130 ) uses a programming library (eg, AR Core, AR Tool Kit, AR SDK) for analyzing real-time computer vision for shading, contrast, point cloud, color, etc. of the image. , Unity, OpenCV (Open Source Computer Vision), Python, etc.), and various algorithms such as SLAM, a depth (eg, a position on the z-axis) can be given to each point (pixel) composing an image. Accordingly, the image acquired through the camera 110 may include 3D position information (eg, an x-axis position, a y-axis position, and a z-axis position).

The stereo camera includes a plurality of the above-described RGB cameras, and the plurality of RGB cameras may be disposed to be spaced apart from each other. Here, the description of the RGB camera may be applied. The stereo camera may acquire a plurality of images by simultaneously photographing a subject at different positions at the same point of time.

In this case, the image processor (or processor 130) calculates disparity by stereo matching a plurality of images, and based on the disparity, the focal length of the lens and the baseline, the camera ( 110) and the depth (or distance) between the subject may be calculated. In addition, the image processor (or processor 130 ) combines two-dimensional position information (eg, an x-axis position, a y-axis position) and depth information (eg, a z-axis position) of a reference image among a plurality of images to obtain a reference image 3D position information (eg, an x-axis position, a y-axis position, and a z-axis position) of an object included in the frame may be acquired. Here, stereo matching refers to matching the same subject included in a plurality of images for the same viewpoint through various methods such as global matching and local matching. The parallax indicates a position difference (eg, a position difference on an x-axis or a y-axis) of the same subject included in a plurality of images, and the greater the focal length or baseline, the higher the parallax. In this case, the focal length may refer to a distance between the image sensor and the lens. The baseline may refer to an interval at which a plurality of RGB cameras are spaced apart. The reference image may refer to an image captured by one preset RGB camera among a plurality of RGB cameras.

In addition, the camera 110 according to an embodiment of the present disclosure may be implemented as an RGB-D (Depth) camera. Here, the RGB-D (Depth) camera may acquire an image by photographing a subject, and detect a depth (or distance) between the camera 110 and the subject. In this case, the processor 140 combines the two-dimensional position information (eg, the x-axis position, the y-axis position) and the depth information (eg, the z-axis position) of the image frame, and the three-dimensional position of the object included in the image frame. Information (eg, position on the x-axis, position on the y-axis, position on the z-axis) can be acquired. To this end, the RGB-D camera may include a sensor (eg, a Time Of Flight (TOF) sensor, a LIDAR sensor, etc.) coupled to an RGB camera or a stereo camera.

According to an embodiment of the present disclosure, the processor 130 may control the display 120 to display an AR object at a specific location on the image while building a map using the SLAM-based tracking technology of the camera 110 . have.

For example, when the camera 110 continuously acquires a plurality of images, the processor 130 constructs a map of a three-dimensional space through a subject, a feature point, or a point cloud of each of the plurality of images. At the same time, the processor 130 compares the map with the three-dimensional space in the image (hereinafter, the current image) most recently acquired through the camera, to the point on the current image corresponding to the point on the map where the AR object is located. The display 120 may be controlled to display the AR object. Accordingly, the point at which the AR object is displayed can be synchronized in real time.

The display 120 is a device for visually outputting information or data. The display 120 may display the image and the AR object in all or part of the display area. The display area may refer to a pixel unit area in which information or data is visually displayed.

Meanwhile, the display 120 may be implemented as a flexible display. In this case, the display 120 has a flexible property that can be bent or bent, and displays images and AR objects even in a bent or bent state. can do. Meanwhile, the display 120 may be implemented as a transparent display, and an object located behind the display 120 may be transmitted through the display 120 due to the transparent nature of the display 120 .

The processor 130 controls the display 120 to display the AR object on the image and the image acquired through the camera 110, and the AR object is displayed on the image while the mode of the electronic device 100 is the normal mode. When the depth of the position is changed, the size of the AR object displayed on the display 120 is adjusted based on the changed depth, and in the state that the mode of the electronic device 100 is the arrangement mode, AR is displayed on the image. When the depth of the point where the object is located is changed, it is possible to control the display 120 to maintain the size at which the AR object is displayed.

More specifically, the processor 130 may control the display 120 to display the AR object on the image and the image acquired through the camera 110 .

Here, the processor 130 may control the display 120 to display the image and the AR object on the image based on the size of the AR object. In this case, the size of the AR object is set to a size displayed at the standard depth for each AR object as a default, and may be changed according to the mode. In addition, the size may collectively refer to various concepts such as area, volume, diagonal length, horizontal and vertical length, radius, and diameter.

In addition, when the depth of the point where the AR object is located on the image is changed while the mode of the electronic device 100 is the normal mode, the processor 130 displays the AR on the display 120 based on the changed depth. You can adjust the size at which the object is displayed.

Here, the depth of the point where the AR object is positioned may be changed as the point where the camera 110 is positioned is moved according to the depth direction. The depth direction may be a z-axis direction.

For example, when the camera 110 moves in the depth direction (when at least one component of the vector with respect to the moving direction of the camera 110 is in the depth direction), the point at which the camera 110 is located may be changed. , due to this, the distance (depth) between the point where the camera 110 is positioned and the point where the AR object is positioned may be changed. This can be applied not only in normal mode but also in batch mode.

According to an embodiment of the present disclosure, the processor 130 performs the display 120 on the basis of the depth of the point where the AR object is located and the attribute assigned to the AR object in a state in which the mode of the electronic device 100 is the normal mode. ), you can adjust the size of the AR object displayed.

Here, the processor 130 displays the display 120 when the depth of the AR object on the image is changed when the attribute assigned to the AR object is a fixed size while the mode of the electronic device 100 is the normal mode. In a state where the size at which the AR object is displayed is maintained, and the mode of the electronic device 100 is in the normal mode, when the property assigned to the AR object is a property whose size changes, the point at which the AR object is located on the image The size at which the AR object is displayed on the display 120 is adjusted so that the size decreases when the depth of the AR object increases, and the size at which the AR object is displayed on the display 120 increases when the depth of the point where the AR object is located decreases. can be adjusted.

Specifically, the attribute assigned to the AR object may be one of a fixed size attribute and a variable size attribute. Each AR object has a property selected according to a user command (eg, a touch gesture, a motion gesture, a user's voice, a mouse click, etc.) through the input interface 170 (refer to FIG. 3) among the properties of fixed size and properties of variable size. can be assigned. Alternatively, even if there is no user command (that is, automatically), the processor 130 may consider that a size-variable attribute is assigned to the AR object when a fixed-size attribute is not assigned to the AR object. Alternatively, the processor 130 may assign a fixed size attribute to the AR object when the AR object is text-type information such as letters, numbers, and symbols, even without a user command (ie, automatically). Here, the attribute assigned to the AR object may be stored in the memory 150 .

Meanwhile, the user command of the present disclosure may be, for example, various types of input such as a touch gesture, a motion gesture, a user's voice, a mouse click, a keyboard input, a button input, and the like, and further, the user command is not limited thereto, and the development of technology is not limited thereto. Accordingly, it can be transformed into any type of input that the user can interact with with the electronic device. Hereinafter, for convenience of description, it is assumed that the touch gesture is described.

Here, in a state in which the mode of the electronic device 100 is the normal mode, the processor 130 displays the AR object on the display 120 based on the changed depth when the property assigned to the AR object is a property whose size changes. You can adjust the displayed size to change. That is, in this case, perspective may be applied to the AR object as shown in FIG. 4A .

Specifically, the processor 130 increases the depth of the point at which the AR object is located on the image when the property assigned to the AR object is a property whose size changes while the mode of the electronic device 100 is the normal mode. If the AR object is adjusted so that the size at which the AR object is displayed on the display 120 is decreased, the AR object is adjusted to decrease the size of the AR object displayed on the display 120 when the depth of the point where the AR object is located on the image is decreased. objects can be adjusted.

For example, it is assumed that the depth of the AR objects 415 and 425 is changed from 1m to 3m by moving the viewpoint of the camera 110 in the normal mode, referring to FIG. 4A .

Here, the processor 130 sets the display 120 to display the AR objects 415 and 425 on the

images

410 and 420 based on the sizes and depths (1m, 3m) of the AR objects 415 and 425 . can be controlled Here, the AR objects 415 and 425 have the same preset size (default size). Also, the size of the AR objects 415 and 425 may be a size displayed at a standard depth (eg, 1 m).

In addition, the processor 130 may control the display 120 to display the AR 415 on the image 410 with a preset size of the AR object 415 at a depth of 1 m, which is a standard depth.

And, when the depth of the AR object is changed to 3m as the point (viewpoint) where the camera 110 is located moves, the processor 130 determines that the property assigned to the AR object is fixed in size and the size is changed. You can determine which of the properties are. Here, when it is determined that the property of the AR object is a property whose size is variable, the processor 130 sets the AR object to a size smaller than the preset size of the AR object 425 by perspective at a depth of 3 m greater than the standard depth ( The display 120 may be controlled to display 425 on the image 420 .

That is, as the depth increases from 1 m to 3 m, the processor 130 may decrease the size of the AR objects 415 and 425 displayed on the

images

410 and 420 . Alternatively, when the depth is decreased from 3m to 1m, the processor 130 may increase the size of the AR objects 415 and 425 displayed on the

images

410 and 420 according to the decreased depth.

On the other hand, the processor 130 controls the display 120 to maintain the size at which the AR object is displayed when the property assigned to the AR object is a fixed size property while the mode of the electronic device 100 is the normal mode. can do. That is, in this case, the perspective for the AR object may be ignored as shown in FIG. 4B .

Specifically, the processor 130 changes the depth of the point at which the AR object is located on the image when the property assigned to the AR object is a property in which the size is fixed while the mode of the electronic device 100 is the normal mode. Even if it is, it is possible to control so that the size of the AR object displayed on the display 120 is maintained.

For example, it is assumed that the depth with the AR objects 435 and 445 is changed from 1m to 3m by moving the viewpoint of the camera 110 in the normal mode, referring to FIG. 4B .

Here, the processor 130 sets the display 120 to display the AR objects 435 and 445 on the

images

430 and 440 based on the sizes and depths (1m, 3m) of the AR objects 435 and 445 . can be controlled Here, the AR objects 435 and 445 have the same preset size (default size). Also, the size of the AR objects 435 and 445 may be a size displayed at a standard depth (eg, 1 m).

In addition, the processor 130 may control the display 120 to display the AR 435 on the image 420 with a preset size of the AR object 435 at a depth of 1 m, which is a standard depth.

In addition, when the depth of the AR object is changed to 3m as the point (viewpoint) at which the camera 110 is located moves in the normal mode, the processor 130 determines that the property assigned to the AR object has a fixed size and size. It is possible to determine which of the properties that is changed. Here, when it is determined that the property of the AR object has a fixed size, the processor 130 sets the AR object 445 to a size smaller than the preset size of the AR object 445 at a depth of 3 m greater than the standard depth. In order to prevent the display from being displayed, the preset size of the AR object 445 may be largely adjusted according to the depth of the point where the AR object 445 is disposed. Accordingly, perspective according to the depth of the AR object may be ignored.

For example, the processor 130 adjusts the preset size (default size) of the AR object according to a formula such as size(B) = size(A) * distance(AB), so that the AR object is displayed on the display. size can be maintained. Here, size(A) is the size when the AR object is located at point A, size(B) is the size when the AR object is located at point B, and distance(AB) is the depth between point A and point B It can represent a difference (or distance).

That is, the processor 130 may maintain the size at which the AR object is displayed on the display 120 at a constant size by adjusting a predetermined original size (or the original size of the AR object) of the AR object.

As described above, in the electronic device 100 according to an embodiment of the present disclosure, even in the normal mode, when the depth of the point at which the AR object is disposed is large or small (ie, when the AR object is far away from or close to the camera 110 ) ), the size of the AR object displayed on the display 120 may be maintained so that the size displayed on the display 120 is not displayed large or small. Accordingly, the visibility of information may be improved in that the user can view an AR object to which an attribute having a fixed size, such as text-type information, is assigned regardless of a depth (distance) in an appropriate size. In addition, user convenience may be improved in that the user does not need to separately adjust the size of the AR object.

Then, when the depth of the point where the AR object is located on the image is changed while the mode of the electronic device 100 is the arrangement mode, the processor 130 controls to maintain the size at which the AR object is displayed on the display 120 . can do. That is, in this case, the perspective for the AR object may be ignored.

On the other hand, in the arrangement mode, not only the editing mode for correcting (editing) the point where the AR object displayed on the image is located, but also an additional mode for locating the AR object not displayed on the image at a specific point on the image. may include

Here, the depth of the point at which the AR object is located may be changed according to the movement of the point at which the camera 110 is located or the point at which the AR object is located on the image according to the depth direction.

For example, when the AR object moves in the depth direction according to a user command (when at least one component of a vector with respect to the moving direction of the AR object is in the depth direction), the point at which the AR object is located on the image may be changed. Thereby, the distance (depth) between the point where the camera 110 is positioned and the point where the AR object is positioned may be changed. This can be applied only in the arrangement mode where the point at which the AR object is placed can be changed.

According to an embodiment of the present disclosure, when the depth of the point where the AR object is located on the image is changed in the arrangement mode, the processor 130 maintains the size at which the AR object is displayed on the display 120 to maintain the AR object. can change the size of

Specifically, when the depth of the point at which the AR object is located increases while the mode of the electronic device 100 is the arrangement mode, the processor 130 controls the size of the AR object to maintain the size at which the AR object is displayed on the display 120 . size can be increased.

Contrary to this, when the depth of the point at which the AR object is located decreases while the mode of the electronic device 100 is the arrangement mode, the processor 130 controls the size of the AR object to maintain the size at which the AR object is displayed on the display 120 . size can be reduced.

For example, it is assumed that the depth with the AR objects 435 and 445 is changed from 1 m to 3 m by moving the point where the AR object is located in the arrangement mode with reference to FIG. 4B .

Here, the processor 130 displays the AR objects 435 and 445 on the

images

430 and 440 based on the sizes and depths (eg, 1m, 3m) of the AR objects 435 and 445 on the display 120 to display the AR objects 435 and 445 . ) can be controlled. Here, the sizes of the AR objects 435 and 445 may be changed from the same preset size (default size).

In addition, the processor 130 may control the display 120 to display the AR 435 on the image 430 with a preset size of the AR object 435 at a depth of 1 m, which is a standard depth.

In addition, when the depth is changed to 3m as the point at which the AR object is located moves according to a user command in the arrangement mode, the processor 130 determines that at a depth of 3m greater than the standard depth, the AR object 445 is larger than the preset size. In order to prevent the AR object 445 from being displayed in a small size, the preset size of the AR object 445 may be increased according to the depth of the point where the AR object 445 is disposed. Accordingly, perspective according to the depth of the AR object may be ignored.

That is, the processor 130 may maintain the size at which the AR object is displayed on the display 120 at a constant size by adjusting a predetermined unique size of the AR object.

As described above, the electronic device 100 according to an embodiment of the present disclosure may maintain the size at which the AR object is displayed on the display 120 so as to ignore the perspective while changing the position at which the AR object is disposed in the arrangement mode. have. Accordingly, user convenience in manipulating the AR object may be improved in that the size of the AR display does not change depending on the depth of the point where the AR object is located, and it is easier for the user to know what the actual size of the AR object is. It has a predictable effect.

Referring to FIG. 3 , the electronic device 100 according to an embodiment of the present disclosure includes a speaker 140 , a memory 150 , a communication unit 160 , and a camera 110 , a display 120 , and a processor 130 in addition to the processor 130 . At least one of the input interfaces 170 may be further included.

The processor 130 may control the electronic device 100 by executing at least one instruction stored in the memory 150 . To this end, the processor 130 may be connected to the camera 110 , the display 120 , and the memory 150 to control the electronic device 100 . Specifically, the processor 130 may read and interpret the instructions and determine a sequence for data processing, thereby controlling the operation of another device by providing timing and control signals for controlling the operation of the other device to the other device. have.

Meanwhile, the processor 130 may control the electronic device 100 by executing at least one instruction stored in a memory (not shown) provided in the processor 130 . In this case, the memory provided in the processor 130 includes a ROM (eg, NOR or NAND type flash memory), a RAM (eg, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM)), volatile memory and the like.

Meanwhile, the processor 130 may include one or a plurality of processors, and the processor 130 is a general-purpose processor such as a central processing unit (CPU), an application processor (AP), a graphic processing unit (GPU), and a vision (VPU). It may be implemented with a graphics-only processor, such as a processing unit, or an artificial intelligence-only processor, such as a neural processing unit (NPU).

Here, the processor 130 may include a GPU and a CPU, and the GPU and the CPU may perform the operations of the present disclosure in connection. For example, the GPU may process an image frame among data, and the CPU may process the remaining data (eg, instructions, code, etc.). At this time, the GPU is implemented in a structure with hundreds or thousands of cores specialized for a parallel processing method that processes multiple commands or data simultaneously, and the CPU has several cores specialized for a serial processing method that processes commands or data in the order in which they are input. It can be implemented with a structure with

As an embodiment, the GPU detects a plurality of fingers from the plurality of first image frames acquired through the camera 110 , and the CPU determines that the detected poses of the plurality of fingers correspond to the trigger poses, inputting a character mode, the GPU detects the motion of one of the plurality of fingers in the plurality of second image frames acquired through the camera 110 in the text input mode, and the CPU detects the position of the finger by the motion and the finger The display 120 may be controlled to identify a key corresponding to a motion among a plurality of keys mapped to a finger based on the position of the set reference point and display information corresponding to the identified key. This is just one embodiment and various modified embodiments are possible, and hereinafter, for convenience of description, it will be described as being performed by the processor 130 .

The speaker 140 may directly output various notification sounds or voice messages as well as various audio data on which various processing operations such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown), and the electronic device ( 100) or may be implemented as a separate external device. Here, the speaker 160 may be implemented as a directional speaker that transmits sound only to a specific location or area.

The memory 150 may refer to hardware that stores information such as data in an electrical or magnetic form so that the camera 110, the processor 130, etc. can access it, and the memory 150 is a non-volatile memory, a volatile memory, and the like. , a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), RAM, ROM, etc. may be implemented as hardware.

At least one instruction, program, or data required for the operation of the electronic device 100 or the processor 130 may be stored in the memory 150 . Here, the instruction is a code unit for instructing the operation of the electronic device 100 or the processor 130 , and may be written in machine language, which is a language that a computer can understand. A program may be a set of instructions that perform a specific task in a unit of work. The data may be status information in units of bits or bytes that can represent characters, numbers, images, and the like.

Also, the memory 150 may store an image frame acquired by the camera 110 , information corresponding to a key identified by the processor 140 , and the like. The memory 150 is accessed by the processor 130 , and reading/writing/modification/deletion/update of instructions, programs, or data may be performed by the processor 130 .

The communication unit 160 may transmit/receive various types of data to/from various types of external devices (eg, servers) according to various wired or wireless communication methods. In this case, the communication unit 160 may perform direct communication with an external device or may communicate with an external device via (or relay) other external devices through various communication networks. To this end, the communication unit 160 may include circuitry according to each communication method, and may further include an antenna or the like in the case of a wireless communication method.

The communication unit 160 may receive information from an external device and transmit the received information to the processor 130 . Also, the communication unit 160 may transmit information to an external device under the control of the processor 130 .

The communication unit 160 is a Wi-Fi chip using a Wi-Fi (Wi-Fi) communication method, a Bluetooth chip using a Bluetooth communication method, an NFC chip using a near field communication (NFC) communication method, a mobile communication method (eg : Infrared communication using a wireless communication chip and infrared communication method using long-term evolution (LTE), LTE Advance (LTE-A), 5th generation (5G), code division multiple access (CDMA), wideband CDMA (WCDMA)) It may include at least one of the chips. Furthermore, the communication unit 160 may include at least one of an Ethernet module (not shown) and a USB module (not shown) for performing wired communication. Here, the communication unit 160 may include a network interface or a network chip according to a wired/wireless communication method. On the other hand, the communication method of the communication unit 160 is not limited to the above-described example, and may include a communication method that appears newly according to the development of technology.

The input interface 170 may receive various user inputs and transmit them to the processor 140 . The input interface 170 may include, for example, at least one of a touch panel (not shown), a pen sensor (not shown), a key (not shown), and a microphone (not shown). The touch panel may use, for example, at least one of a capacitive type, a pressure-sensitive type, an infrared type, and an ultrasonic type, and for this, the touch panel may include a control circuit. The touch panel may further include a tactile layer to provide a tactile response to the user. The pen sensor may be, for example, a part of the touch panel or may include a separate recognition sheet. The key may include, for example, a physical button, an optical key, or a keypad. The microphone may directly receive the user's voice, and digitally convert the user's voice, which is an analog signal, by a digital converter (not shown) to obtain an audio signal. Such an input interface 170 may be embedded in the electronic device 100 or implemented as a separate external device (not shown) such as a keyboard, mouse, external microphone, remote control, or the like.

5 to 11 are diagrams for explaining a method of adding an AR object to an image according to an embodiment of the present disclosure.

Referring to FIG. 5 , the processor 130 of the electronic device 100 may control the display 120 to display an image 520 acquired through the camera 110 . In this case, the image 520 may include a subject photographed through the camera 110 , where the subject may include a real environment 510 (space or object) located within the angle of view from the viewpoint of the camera 110 . can

In this case, the processor 130 may recognize each of a plurality of pixels included in the image 520 as 3D spatial coordinates including a depth by using the SLAM-based tracking technology of the camera 110 . That is, the processor 130 may recognize the image as a three-dimensional space. Furthermore, the processor 130 may generate a plurality of images 520 sequentially acquired through the camera 110 as a map of a three-dimensional space and store it in the memory 150 .

In addition, the processor 130 may control the display 120 to display

UIs

530 and 550 for selecting the arrangement mode. Here, when a user command to select the first and

second UIs

530 and 550 is received, the mode of the electronic device 100 may be set to the arrangement mode.

Here, when a user command for selecting a position where the first UI 530 is displayed as shown in FIG. 5 is received, the processor 130 sets the mode of the electronic device 100 as an additional mode among the arrangement modes as shown in FIG. 6 , A list of a plurality of pre-stored AR objects 631-633 may be displayed on the display 120 . Meanwhile, when a user command for selecting a position where the second UI 550 is displayed in FIG. 5 is received, the processor 130 sets the mode of the electronic device 100 to the edit mode among the batch modes, and sets the image according to the user command. The position of one AR object among the AR objects displayed on the image may be changed. Specific details thereof will be described later with reference to FIGS. 12 to 13C .

In this case, the plurality of AR objects 631 - 633 are AR objects pre-stored in the memory 140 , and may be a rendered image or a text box capable of inputting text. For example, AR objects are objects such as TVs, digital picture frames, sound bars, refrigerators, washing machines, furniture, cars, buildings, trees, etc. that are rendered in 2D or 3D images, text, text, images, photos, videos, etc. It may be in the form of various information such as documents, dashboards, and the like.

Referring again to FIG. 6 , when a user command for selecting a third AR object 633 from a list of a plurality of AR objects 631-633 is received, the processor 130 receives a third AR object as shown in FIG. 7 . The display 120 may be controlled to display 733 on the image 720 . Here, the added AR object 733 may be displayed with a preset size. That is, the perspective for the AR object 733 may be ignored.

In this case, referring to FIG. 7 , the AR object 733 displayed on the display 120 may be a text box, and the AR object 733 may include text input according to a user command.

Meanwhile, referring to FIG. 8 , the processor 130 may change a point at which the AR object 833 added on the image 820 is located in the addition mode according to a user command (eg, touch drag, etc.). In this case, the point at which the AR object 833 is located may be moved along the 3D space axis (x, y, z axis) in the image 820 according to a user command. Here, even when the depth of the AR object 833 is changed by moving the AR object 833 in the z-axis direction (depth direction), perspective may be ignored.

Meanwhile, referring to FIG. 9 , the processor 130 may change the preset size of the AR object 933 added to the image 920 in the addition mode according to a user command (eg, pinch zoom). In this case, the size of the AR object 933 displayed on the display 120 may be changed according to a user command.

And, referring to FIG. 10 , when a user command (eg, a touch longer than a preset time) for the AR object 1033 added to the image 1020 is received in the addition mode, the processor 130 receives the AR object 1033 . ), the display 120 may be controlled to display a UI 1035 for allocating properties and a UI 1037 for deleting the AR object 1033 .

Here, when a user command for selecting a UI 1035 for allocating a property for the AR object 1033 is received, the processor 130 sets the property for the AR object 1033 to a fixed size property and a variable size property. A UI for selecting one of the properties may be controlled to be displayed on the display 120 . Here, the attribute assigned to the AR object may be changed according to a user command.

Alternatively, when a user command for selecting the UI 1037 for deleting the AR object 1033 is received, the processor 130 may remove (or delete) the AR object 1033 displayed on the image 1020 . .

Meanwhile, referring to FIGS. 7 to 11 , the processor 130 sets the AR objects 733 , 833 , 933 , 1033 , 1133 added according to a user command to a set state (size, location, properties, etc.) in the normal mode. The first UI (740, 840, 940, 1040, 1140) that saves the state set to be displayed and the AR object (733, 833, 933, 1033, 1133) added according to the user's command can be canceled from being displayed in normal mode. The display 120 may be controlled to display the

second UIs

770 , 870 , 970 , 1070 , and 1170 .

Here, the processor 130 enters the mode of the electronic device 100 into the normal mode according to a user command to select the

first UI

740 , 840 , 940 , 1040 , 1140 , and an AR object ( The display 120 may be controlled to display 733 , 833 , 933 , 1033 , and 1133 according to a set state.

In contrast, the processor 130 enters the mode of the electronic device 100 into the normal mode according to a user command for selecting the

second UI

770 , 870 , 970 , 1070 , and 1170 , and the AR object 733 in the normal mode , 833 , 933 , 1033 , 1133 may be removed (or deleted) from the AR objects 733 , 833 , 933 , 1033 , and 1133 so that they are not displayed.

12 to 13C are diagrams for explaining an arrangement mode according to an embodiment of the present disclosure.

Referring to FIG. 12 , the processor 130 of the electronic device 100 may control the display 120 to display

UIs

1230 and 1250 for selecting a layout mode.

Here, when a user command for selecting the first UI 1230 is received, the processor 130 may set (or enter) the mode of the electronic device 100 as an additional mode from among the batch modes, and a description thereof is shown in FIG. 5 . It has been described above with reference to FIGS.

On the contrary, when a user command for selecting the second UI 1250 is received, the processor 130 may set (or enter) the mode of the electronic device 100 from the arrangement mode to the edit mode. In this case, the processor 130 may change the arrangement of the AR object 1233 among the image 1220 and the AR object 1233 displayed on the display 120 according to a user command.

13A to 13C , in the arrangement mode, points located on the

images

1320A and 1320B may be changed according to a user command of the AR objects 1333A and 1333B.

For example, as shown in FIGS. 13A and 13C , when the point where the AR object 1333A is located in the arrangement mode is moved to a point where the depth decreases according to a user command, the processor 130 displays the AR on the image 1320A. The preset size of the AR object 1333A may be decreased so that the displayed size of the object 1333A is maintained.

For another example, as shown in FIGS. 13B and 13C , when the point at which the AR object 1333B is located in the arrangement mode is moved to a point at which the depth increases according to a user command, the processor 130 displays the image 1320B on the image 1320B. The preset size of the AR object 1333B may be increased so that the size at which the AR object 1333B is displayed is maintained.

On the other hand, in the case of the aforementioned FIGS. 13A and 13B , the processor 130 determines the size at which the AR objects 1333A and 1333B are displayed on the display 120 and the AR object 1233 is displayed on the display 120 of FIG. 12 . You can control it to stay at the displayed size. That is, the size of the AR objects 1333A and 1333B displayed on the display 120 may be maintained as the size of the AR object 1233 displayed on the display 120 when the edit mode is set.

As such, in the arrangement mode according to an embodiment of the present disclosure, by changing the intrinsic size of the AR object based on the depth, the effect of changing the size by perspective may be offset.

14 to 15B are diagrams for explaining a normal mode according to an embodiment of the present disclosure.

Referring to FIG. 14 , when the electronic device 100 is set to the normal mode, the processor 130 of the electronic device 100 obtains an image 1420 and

AR objects

1431 and 1432 obtained through the camera 110 . The display 120 may be controlled to display . Here, the AR objects 1431 and 1432 may be arranged on the image 1420 through the above-described arrangement mode.

Here, it is assumed that the electronic device 100 in the state shown in FIG. 14 is moved to a point where the depth is changed as shown in FIGS. 15A and 15B . In this case, it is assumed that the first AR objects 1531A and 1531B are assigned with a variable size attribute, and the second AR objects 1532A and 1532B are assigned with a fixed size attribute.

For example, as shown in FIG. 15A , the first and second AR objects 1531A and 1532A located on the image 1520A obtained by the electronic device 100 having the camera 110 through the camera 110 are shown in FIG. 15A . When the depth is moved in a decreasing direction (ie, when the electronic device 100 approaches the subject), the processor 130 may determine a property assigned to each of the first and second AR objects 1531A and 1532A. have.

In this case, the processor 130 increases the size at which the first AR object 1531A is displayed on the image 1520A as the depth decreases with respect to the first AR object 1531A to which the property whose size is changed is assigned. The first AR object 1531A may be adjusted to

Also, with respect to the second AR object 1532A to which the property to which the size is fixed is assigned, the processor 130 is configured to maintain the size at which the second AR object 1532A is displayed on the image 1520A. The second AR object 1532A may be adjusted to reduce a preset size of the second AR object 1532A.

As another example, as shown in FIG. 15B , the first and second AR objects 1531B and 1532B located on the image 1520B obtained by the electronic device 100 having the camera 110 through the camera 110 , as shown in FIG. 15B . ) is moved in an increasing direction (ie, when the electronic device 100 moves away from the subject), the processor 130 determines a property assigned to each of the first and second AR objects 1531B and 1532B. can

In this case, the processor 130 decreases the size of the first AR object 1531B displayed on the image 1520B as the depth increases with respect to the first AR object 1531B to which the property whose size is changed is assigned. can be adjusted to make it

In addition, the processor 130 controls the size of the second AR object 1532B to which the property to be fixed in size is assigned so that the size at which the second AR object 1532B is displayed on the image 1520B is maintained. It may be adjusted to increase the preset size of the second AR object 1532B.

As described above, according to an embodiment of the present disclosure, perspective is applied to the first AR object 1531A, and perspective is ignored to the second AR object 1532A according to the attribute assigned to the AR object in the normal mode. can

Referring to FIG. 16 , the control method of the electronic device 100 according to an embodiment of the present disclosure includes an image acquired through the camera 110 and displaying an AR object on the image ( S1610 ), the electronic device When the depth of the point where the AR object is located on the image is changed in the state in which the mode of (100) is the normal mode, adjusting the size of the AR object is displayed based on the changed depth (S1620) and the former In a state in which the mode of the device 100 is the arrangement mode, if the depth of the point where the AR object is located on the image is changed, controlling to maintain the size at which the AR object is displayed ( S1630 ) is included.

More specifically, the AR object may be displayed on the image and the image acquired through the camera 110 first (S1610). Here, the AR object may be displayed on the image and the image acquired through the camera 110 based on the size of the AR object. In this case, the size of the AR object is set to a size displayed at the standard depth for each AR object as a default, and may be changed according to the mode. In addition, the size may collectively refer to various concepts such as area, volume, diagonal length, horizontal and vertical length, radius, and diameter.

Thereafter, when the depth of the point where the AR object is located on the image is changed while the mode of the electronic device 100 is the normal mode, the size at which the AR object is displayed may be adjusted based on the changed depth ( S1620).

In this case, the depth of the point where the AR object is positioned may be changed as the point where the camera 110 is positioned is moved according to the depth direction.

Here, in the adjusting step, in the normal mode, the size at which the AR object is displayed may be adjusted based on the depth of the point where the AR object is located and the property assigned to the AR object.

As an embodiment, in the adjusting step, in the normal mode, when the property assigned to the AR object is a property with a fixed size, the size at which the AR object is displayed is maintained, and the size of the property assigned to the AR object is changed. In the case of a variable property, the size at which the AR object is displayed may be adjusted based on the changed depth.

As an embodiment, in the step of adjusting, in the normal mode, if the property assigned to the AR object has a fixed size, and the depth of the AR object on the image is changed, the size at which the AR object is displayed is maintained. can be controlled as much as possible.

As an embodiment, in the adjusting step, in the normal mode, if the property assigned to the AR object is a property whose size changes, the AR object is adjusted so that the size decreases when the depth of the point where the AR object is located on the image increases. The displayed size may be adjusted, and the displayed size of the AR object may be adjusted so that the size increases when the depth of the point where the AR object is located decreases.

Thereafter, when the depth of the point at which the AR object is located on the image is changed in a state in which the mode of the electronic device 100 is the arrangement mode, control may be performed to maintain the size at which the AR object is displayed ( S1630 ).

In this case, the depth of the point at which the AR object is positioned may be changed as the point at which the camera 110 is positioned or the point at which the AR object is positioned on the image is moved according to the depth direction.

Here, in the arrangement mode, if the depth of the point where the AR object is located on the image is changed, the size of the AR object may be changed to maintain the size at which the AR object is displayed.

As an embodiment, the controlling may include increasing the size of the AR object so as to maintain the displayed size of the AR object when the depth of the point where the AR object is located increases in the arrangement mode, and increasing the size of the point where the AR object is located. When the depth decreases, the size of the AR object can be reduced to maintain the size at which the AR object is displayed.

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium readable by a machine (eg, a computer). The device calls the stored instructions from the storage medium. and an electronic device (eg, the electronic device 100) according to the disclosed embodiments as a device capable of operating according to the called command. When the command is executed by the processor, the processor directly or the The function described in the instruction may be performed using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter. A machine-readable storage medium is a non-transitory It may be provided in the form of a (non-transitory) storage medium, where 'non-transitory' means that the storage medium does not include a signal and is tangible, but data is semi-permanent or temporary in the storage medium It does not distinguish that it is stored as

The method according to various embodiments may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as merchandise. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

Each of the components (eg, a module or a program) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-components of the above-described sub-components may be omitted, or other sub-components may be various. It may be further included in the embodiment. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each of the components before being integrated. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallel, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

Claims

In an electronic device,

camera;

display; and

controlling the display to display an image acquired through the camera and an AR object on the image,

When the depth of the point where the AR object is located on the image is changed while the mode of the electronic device is the normal mode, the size of the AR object displayed on the display is adjusted based on the changed depth and,

Electronic device comprising a; if the depth of the point where the AR object is located on the image is changed in a state in which the mode of the electronic device is the arrangement mode, the processor controls to maintain the size at which the AR object is displayed on the display; .
According to claim 1,

The depth of the point where the AR object is located is,

The electronic device is changed as a point at which the camera is positioned or a point at which the AR object is positioned on the image is moved according to a depth direction.
According to claim 1,

The processor is

Control the display to display the AR object on the image and the image based on the size of the AR object,

In the arrangement mode, when the depth of the point where the AR object is located on the image is changed, the size of the AR object is changed to maintain the size at which the AR object is displayed on the display.
4. The method of claim 3,

The processor is

When the depth of the point at which the AR object is located increases while the mode of the electronic device is the arrangement mode, the size of the AR object is increased to maintain the size at which the AR object is displayed on the display, and the AR object is The electronic device of claim 1, wherein when the depth of the location decreases, the size of the AR object is reduced to maintain the size at which the AR object is displayed on the display.
According to claim 1,

The processor is

Control the display to display the AR object on the image and the image based on the size of the AR object,

Adjusting a size at which the AR object is displayed on the display based on a depth of a point where the AR object is located and a property assigned to the AR object in a state in which the mode of the electronic device is a normal mode.
6. The method of claim 5,

The processor is

In a state in which the mode of the electronic device is a normal mode, if the attribute assigned to the AR object is a fixed size attribute, control to maintain the size at which the AR object is displayed on the display, and the attribute assigned to the AR object If the size is a variable property, adjusting the size of the AR object displayed on the display to be changed based on the changed depth.
7. The method of claim 6,

The processor is

In a state in which the mode of the electronic device is the normal mode, when the attribute assigned to the AR object is a fixed size attribute, the size at which the AR object is displayed on the display is changed when the depth of the AR object is changed on the image. control to keep

In a state in which the mode of the electronic device is the normal mode, when the property assigned to the AR object is a property whose size changes, the display is displayed on the display so that the size decreases when the depth of the point where the AR object is located on the image increases. Adjusting a size at which the AR object is displayed, and adjusting a size at which the AR object is displayed on the display so that the size increases when a depth of a point where the AR object is located decreases.
A method for controlling an electronic device, comprising:

displaying an image acquired through a camera and an AR object on the image;

adjusting a display size of the AR object based on the changed depth when the depth of the point where the AR object is located on the image is changed while the mode of the electronic device is the normal mode; and

and controlling to maintain a displayed size of the AR object when the depth of the point at which the AR object is positioned on the image is changed while the mode of the electronic device is the arrangement mode.
9. The method of claim 8,

The depth of the point where the AR object is located is,

A control method that is changed as a point at which the camera is positioned or a point at which the AR object is positioned on the image is moved according to a depth direction.
9. The method of claim 8,

The displaying step is

displaying the AR object on the image and the image based on the size of the AR object,

The controlling step is

In the arrangement mode, if the depth of the point where the AR object is located on the image is changed, changing the size of the AR object so as to maintain the displayed size of the AR object.
11. The method of claim 10,

The controlling step is

increasing the size of the AR object so as to maintain a displayed size of the AR object when the depth of the point at which the AR object is located increases in the arrangement mode; and

In the arrangement mode, if the depth of the point at which the AR object is located decreases, reducing the size of the AR object so as to maintain the displayed size of the AR object.
9. The method of claim 8,

The displaying step is

displaying the AR object on the image and the image based on the size of the AR object,

The adjusting step is

In the normal mode, the control method of adjusting a size in which the AR object is displayed based on a depth of a point where the AR object is located and a property assigned to the AR object.
13. The method of claim 12,

The adjusting step is

controlling the size at which the AR object is displayed to be maintained when the attribute assigned to the AR object has a fixed size in the normal mode; and

In the normal mode, when the property assigned to the AR object is a property whose size changes, adjusting the displayed size of the AR object to change based on the changed depth.
14. The method of claim 13,

The adjusting step is

controlling the size at which the AR object is displayed to be maintained when the depth of the AR object is changed on the image when the attribute assigned to the AR object is a fixed size in the normal mode; and

In the normal mode, when the property assigned to the AR object is a property whose size changes, the size at which the AR object is displayed is determined so that the size decreases when the depth of the point where the AR object is located on the image increases. and adjusting a size at which the AR object is displayed so that the size increases when the depth of the point at which the AR object is located decreases.