KR102640871B1 - Electronic apparatus for providing image data by using augmented reality and control method thereof - Google Patents

Abstract

In this disclosure, an electronic device and a method for controlling the same are provided.
The electronic device of the present disclosure determines a boundary area between a wall area and an area other than the wall area in a sensor, a display, a camera, and an image frame acquired by the camera, and determines the location of the electronic device determined by the sensor and the determined boundary area. Based on this, a processor determines the distance between the wall corresponding to the wall area and the electronic device, and controls the display so that an object is displayed in the wall area with a size corresponding to the determined distance.

Description

Electronic device providing image data using augmented reality and control method thereof {ELECTRONIC APPARATUS FOR PROVIDING IMAGE DATA BY USING AUGMENTED REALITY AND CONTROL METHOD THEREOF}

This disclosure relates to an electronic device and method for providing image data using augmented reality, and more specifically, to an electronic device for providing image data including augmented reality information and a method for controlling the same.

With the development of electronic technology, various electronic devices are being developed. Recently, electronic devices incorporating augmented reality (AR) or mixed reality (MR) are being commercialized. Here, augmented reality or mixed reality (hereinafter collectively referred to as augmented reality) can synthesize virtual objects into a real environment to make them look like objects that exist in reality, or provide additional information necessary for the real environment. .

In particular, augmented reality is used not only in industrial fields such as manufacturing and marketing, but also in everyday life, allowing users to place virtual objects of a specific size like real objects (or products) on images taken of real spaces. there is. At this time, the farther the distance to the space where the object is placed, the smaller the size of the object appears, so technology that can detect the plane on which the object is positioned and accurately measure the distance to the plane is required.

Meanwhile, in the case of images taken using dual lenses, the distance can be measured by obtaining 3D information from images in which each lens simultaneously captures one subject from different angles. However, in the case of a single lens (or single lens, hereinafter referred to as a single lens), there is a structural limitation in that the dual lens method cannot be adopted in that there is only one lens.

Additionally, in the case of images taken of a monochromatic wall using a single lens, there is a problem in that the monochromatic wall without texture or pattern cannot be detected as a vertical plane, making it impossible to measure the exact distance to the wall.

The present disclosure was created in response to the above-described need, and the purpose of the present disclosure is to provide an electronic device capable of detecting a wall area in an image frame and providing a composite image of an object for augmented reality on the wall area in real time, and the same. It provides a control method.

In order to achieve the above object, an electronic device according to an embodiment of the present disclosure determines a boundary area between a wall area and an area other than the wall area in a sensor, a display, a camera, and an image frame acquired by the camera, and Based on the location of the electronic device determined and the determined boundary area, the distance between the wall surface corresponding to the wall area and the electronic device is determined, and the display is controlled so that an object is displayed in the wall area with a size corresponding to the determined distance. May include a processor.

Here, in each of the plurality of image frames acquired by the camera, the processor determines a reference pixel existing at a preset position and a similar pixel having color similarity greater than or equal to a preset threshold, and determines the reference pixel and similarity between the plurality of image frames. If the difference in size of the area containing the pixel is within a preset threshold range, the area containing the reference pixel and the similar pixel may be determined to be the wall area.

Meanwhile, the processor determines a pattern area containing a specific pattern in an area other than the wall area, obtains a set of three-dimensional points in the pattern area based on the position of the electronic device determined by the sensor, and obtains a set of three-dimensional points. The boundary area can be judged according to the straight line corresponding to .

Meanwhile, the processor determines the extent to which the position of the wall area changes in a plurality of image frames acquired by the camera while the electronic device is moving, and determines the extent to which the position of the wall area changes and the position of the electronic device determined by the sensor. Based on the degree of change, the distance between the boundary corresponding to the boundary area and the electronic device can be determined.

Meanwhile, the area other than the wall area includes a ceiling area existing above the wall area and a floor area existing below the wall area in the image frame, and the processor determines a first boundary area between the ceiling area and the wall area in the image frame. And a second boundary area between the floor area and the wall area can be determined.

Here, the processor may determine the distance between the electronic device and the wall corresponding to the wall area based on the distance between the electronic device and the boundary area and the height of the electronic device obtained by the sensor.

Here, the processor determines the height of the electronic device relative to the floor based on the position of the electronic device determined by the sensor, the distance between the electronic device and the boundary corresponding to the second boundary area, and the height of the electronic device. Based on this, the distance between the wall and the electronic device can be determined.

Meanwhile, the processor may determine a size corresponding to the determined distance based on the preset size and preset distance of the object, and control the object to be displayed on the wall area at the determined size.

Meanwhile, when an input for selecting a wall area is received, the processor may control the display to display an object at a selected location in the wall area based on the received input.

Meanwhile, a control method of an electronic device according to an embodiment of the present disclosure for achieving the above object includes determining a boundary area between a wall area and an area other than the wall area in an image frame acquired by a camera, a sensor Based on the location of the electronic device determined by and the determined boundary area, determining a distance between the electronic device and a wall surface corresponding to the wall area and displaying an object in the wall area with a size corresponding to the determined distance. may include.

Here, the step of determining the border area includes determining a similar pixel having a color similarity greater than a preset threshold value to a reference pixel existing at a preset position in each of the plurality of image frames acquired by the camera, and If the difference in size of the area including the reference pixel and the similar pixel between frames is within a preset threshold range, the step of determining the area including the reference pixel and the similar pixel as the wall area may be further included.

Meanwhile, the step of determining the boundary area includes determining a pattern area including a specific pattern in an area other than the wall area, obtaining a set of three-dimensional points in the pattern area based on the position of the electronic device determined by the sensor, and , the boundary area can be determined according to a straight line corresponding to a set of 3D points.

Meanwhile, the step of determining the distance between the wall and the electronic device includes determining the degree to which the position of the wall region changes in a plurality of image frames acquired by the camera while the electronic device is moving, and the degree to which the position of the wall region changes. and determining the distance between the boundary corresponding to the boundary area and the electronic device based on the degree to which the position of the electronic device determined by the sensor changes.

Meanwhile, the area other than the wall area includes a ceiling area that exists above the wall area in the image frame and a floor area that exists below the wall area, and the step of determining the boundary area includes the ceiling area and the wall area in the image frame. A first boundary area between and a second boundary area between the floor area and the wall area can be determined.

Here, the step of determining the distance between the wall and the electronic device includes the distance between the wall and the electronic device corresponding to the wall area based on the distance between the electronic device and the boundary area and the height of the electronic device obtained by the sensor. can be judged.

Here, the step of determining the distance between the wall surface and the electronic device includes determining the height of the electronic device relative to the floor based on the position of the electronic device determined by the sensor, and determining the height of the electronic device based on the floor surface and determining the distance between the wall and the electronic device. And the distance between the wall and the electronic device can be determined based on the distance between the electronic devices and the height of the electronic device.

Meanwhile, in the displaying step, the size corresponding to the determined distance may be determined based on the preset size and preset distance of the object, and the object may be displayed on the wall area at the determined size.

Meanwhile, in the displaying step, when an input for selecting a wall area is received, an object may be displayed at a location selected from the wall area based on the received input.

According to various embodiments of the present disclosure as described above, an electronic device that detects a wall area in an image and provides an image combining objects for augmented reality in the wall area in real time and a control method thereof can be provided.

In addition, when measuring the distance between an electronic device and a monochromatic wall using an image frame captured by a single lens, the accuracy of the distance measurement can be improved, and the product can be miniaturized and simplified compared to the case of having a dual lens. and can reduce production costs.

FIG. 1 is a diagram for explaining an electronic device according to an embodiment of the present disclosure.
Figure 2 is a block diagram of an electronic device according to an embodiment of the present disclosure.
3 to 6 are diagrams for explaining a control method of an electronic device according to an embodiment of the present disclosure.
FIG. 7 is a diagram for explaining an electronic device according to another embodiment of the present disclosure.
FIG. 8 is a block diagram illustrating in detail the configuration of an electronic device according to an embodiment of the present disclosure.
FIG. 9 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the following examples may be modified into various other forms, and the scope of the technical idea of the present disclosure is not limited to the following examples. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to completely convey the technical idea of the present disclosure to those skilled in the art.

The technology described in this disclosure is not intended to be limited to specific embodiments, but should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of this disclosure. In connection with the description of the drawings, similar reference numbers may be used for similar components.

Expressions such as “first,” “second,” “first,” or “second,” used in the present disclosure can modify various components regardless of order and/or importance, and can refer to one component. It is only used to distinguish from other components and does not limit the components.

In the present disclosure, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” includes (1) at least one A, (2) at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

In this disclosure, singular expressions include plural expressions, unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “consist of” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are intended to indicate the presence of one or more other It should be understood that this does not exclude in advance the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), It may be understood that no other component (e.g., a third component) exists between other components.

The expression “configured to” used in the present disclosure may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. ," can be used interchangeably with "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Hereinafter, various embodiments of the present disclosure will be described in detail using the attached drawings.

1 is a diagram for explaining an electronic device according to an embodiment of the present disclosure.

As shown in FIG. 1, the electronic device 100 is a device capable of providing augmented reality and may be implemented as a smartphone.

The electronic device 100 may provide the user with an image in which objects for augmented reality are arranged in an image taken of a real space so that the user can intuitively understand the space occupied by the object.

Specifically, the electronic device 100 may generate an image in which an object for augmented reality overlaps an image of a space including a monochromatic wall 10 and provide the image to the user in real time. At this time, the monochromatic wall surface 10 refers to a vertical plane with a single color without texture or pattern.

To this end, the electronic device 100 determines a monochromatic wall area in an image frame acquired by the camera 130, which photographs a space including a monochromatic wall 10 using a single lens, and corresponds to the wall area. The distance between the wall surface and the electronic device is determined, and the object for augmented reality is captured by an image frame acquired by the camera 130 based on the size corresponding to the distance between the wall surface corresponding to the wall area and the electronic device 100. It can be displayed on the wall area inside.

At this time, the object for augmented reality may be a three-dimensional rendering of a real object. For example, an object may be implemented as a three-dimensional rendered image of an object that can be attached and placed on a wall, such as a wall-mounted TV, digital picture frame, or sound bar. However, this is only an example, and is not limited to this, and may be implemented in various modifications. Below, for convenience of explanation, the object is assumed to be a rendered image of a wall-mounted TV.

Accordingly, even when photographing a monochromatic wall 10 using a single lens, the electronic device 100 determines the position, height, and size (for example, 50 inches) of a wall-mounted TV to be placed on the monochromatic wall 10. , 60 inches, 70 inches), etc. can be provided to users with augmented reality that allows them to compare them easily and conveniently.

Meanwhile, in FIG. 1, the electronic device 100 is shown as a smartphone, but this is only an example, and the electronic device 100 may include a tablet PC, speaker, mobile phone, telephone, e-book reader, desktop PC, laptop PC, workstation, Servers, PDAs, PMPs (portable multimedia players), MP3 players, medical devices, cameras, televisions, DVDs (digital video disk) players, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set-top boxes, home automatics. Devices such as electronic control panels, security control panels, media boxes (e.g. Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game consoles (e.g. XboxTM, PlayStationTM), electronic dictionaries, electronic keys, camcorders, electronic picture frames, or wearable devices. It can be implemented as:

Hereinafter, an electronic device that provides augmented reality and a control method thereof will be described in more detail with reference to the drawings.

Figure 2 is a block diagram of an electronic device 100 according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 may include a sensor 110, a display 120, a camera 130, and a processor 140.

The sensor 110 can detect the angle, height, motion (moving and/or rotating speed, direction, distance, etc.) of the electronic device 100. At this time, the sensor 110 may include at least one of an acceleration sensor, a gyro sensor, a proximity sensor, a geomagnetic sensor, a gravity sensor, and a pressure sensor, and may include a motion sensor combining these sensors.

The sensor 110 may detect the amount of light (or brightness) of the surrounding environment of the electronic device 100, and in this case, the sensor 110 may include an optical sensor such as an illuminance sensor.

The sensor 110 may include a beam sensor such as a laser sensor, an infrared sensor, etc. In this case, the sensor 110 may directly detect the distance between the electronic device 100 and the surrounding space.

The display 120 may display image data processed by an image processor (not shown) on a display area (or display). The display area may refer to at least a portion of the display 120 exposed on one side of the housing of the electronic device 100.

At least a portion of the display 120 may be coupled to at least one of the front area, side area, and rear area of the electronic device 100 in the form of a flexible display. Flexible displays can be characterized by being able to bend, bend, or roll without damage through a paper-thin, flexible substrate.

The display 120 may be implemented as a touch screen with a layered structure by combining with a touch panel (not shown). The touch screen may have not only a display function but also a function of detecting not only the location of the touch input, the area of the touch input, but also the pressure of the touch input. In addition, not only the touch that actually touches the touch screen (real-touch) but also the touch screen It may also have the function of detecting proximity touch.

The camera 130 may acquire an image frame for the space by photographing the space using a single lens. At this time, the space may be a space including a wall. Thereafter, the image frame acquired by the camera 130 may be processed by an image processor (not shown) and displayed on the display 120.

The processor 140 (or control unit) may control the overall operation of the electronic device 100.

The processor 140 may determine the boundary area between the wall area and areas other than the wall area in the image frame acquired by the camera 130.

To this end, the processor 140 may determine a monochromatic wall area in the image frame acquired by the camera 130. At this time, the image frame may be acquired by the camera 130 using a single lens, and may be a plurality of image frames acquired in real time by continuously photographing a space including a monochromatic wall.

Specifically, the processor 140 may determine, in each of the plurality of image frames acquired by the camera 130, a reference pixel existing at a preset position and a similar pixel having color similarity greater than or equal to a preset threshold. At this time, the reference pixel may be set to a pixel located at the center of the image frame. Here, the preset position or preset threshold is a preset value or a value set by the user, and can be changed by the user.

In this case, the processor 140 compares the difference between the color value of the pixel located in the center and the color values of the remaining pixels, based on the color value (e.g., R, G, B) of the pixel located in the center of the image frame. Therefore, it can be determined that the smaller the difference in color values, the higher the similarity. Additionally, the processor 140 may determine a pixel having a similarity greater than or equal to a preset threshold as a similar pixel. Here, the preset threshold value is a preset value or a value set by the user and can be changed by the user.

Additionally, if the difference in size of the area including the reference pixel and the similar pixel between the plurality of image frames is within a preset threshold range, the processor 140 may determine the area including the reference pixel and the similar pixel to be the wall area. . Here, the preset threshold range is a preset value or a value set by the user, and can be changed by the user.

In this case, the processor 140 determines the area containing the reference pixel and the similar pixel for each of the plurality of image frames, and determines the size (or position, etc.) of the area containing the reference pixel and the similar pixel between the plurality of image frames. Compare the difference, and if the size difference is within a preset threshold range (i.e., if the size of the area is kept constant within the preset threshold range in a plurality of image frames), the area containing the reference pixel and similar pixels can be judged as the wall area.

Accordingly, even when the color of a pixel may temporarily change depending on the brightness of light or reflection of light in the surrounding environment, the processor 140 can more accurately detect a monochromatic wall area.

Meanwhile, in the case of an image frame acquired in a dark environment (i.e., a low-light environment), the color value of the pixel may be distorted, so the processor 140 determines that the illuminance value detected by the sensor 110 is set to a preset value (e.g. : 3 lx) or more, the operation of detecting a monochromatic wall area in an image frame can be performed. Here, the preset value is a preset value or a value set by the user, and can be changed by the user.

Meanwhile, the image frame may be directly acquired by the camera 130 using a single lens included in the electronic device 100, but this is only an example, and the electronic device 100 may be used to capture images taken by an external device using a single lens. Image frames can also be received from an external device.

Additionally, the processor 140 may determine the boundary area between the wall area and areas other than the wall area in the image frame. At this time, areas other than the wall area may have a specific texture or pattern that is distinct from the wall area, or may be a solid color area.

Specifically, the processor 140 may determine a pattern area including a specific pattern in an area other than the wall area for each of the plurality of image frames. At this time, a pattern may refer to a set of pixels that form a specific shape or a set of pixels that are repeated according to a certain rule. For example, the pattern area may be a shadow area, an invisible area, etc.

Here, the processor 140 may determine a pattern area containing a specific pattern only for the wall area and an adjacent area having a preset threshold range among other areas. Accordingly, the speed and accuracy of determining the pattern area can be improved by reducing the calculation range of the processor 140.

Next, the processor 140 may determine the distance between the electronic device 100 and a wall corresponding to the wall area based on the location of the electronic device 100 determined by the sensor 110 and the determined boundary area. there is. At this time, the wall surface (real space) may correspond to the wall area (virtual space) in the image frame.

To this end, the processor 140 may determine the distance between the boundary and the electronic device 100 based on the movement of the electronic device 100 determined by the sensor 110 and corresponding to the boundary area. At this time, the boundary (real space) may correspond to the boundary area (virtual space) between the wall area and areas other than the wall area in the image frame.

Additionally, the processor 140 may obtain a set of three-dimensional points in the pattern area based on the movement of the electronic device 100 determined by the sensor 110 in each of the plurality of image frames.

Specifically, when the electronic device 100 rotates, the processor 140 may determine the angle at which the image frame is captured according to rotation information of the electronic device 100 determined by the sensor 110. Thereafter, the processor 140 may obtain a depth image by performing stereo synthesis on two image frames sequentially photographed at different angles. Thereafter, the processor 140 may obtain a set of 3D points in the pattern area of the depth image on which stereo synthesis was performed.

At this time, the set of 3D points may have attribute values such as position (or vector) information on the X, Y, and Z axes, reflection intensity, etc., and may be implemented like a point cloud.

Additionally, the processor 140 may determine the boundary area according to a straight line corresponding to a set of 3D points. For example, the processor 140 may determine that a straight line generated according to an algorithm for linearizing a set of 3D points is a boundary area.

Meanwhile, areas other than the wall area may include a ceiling area that exists above the wall area and a floor area that exists below the wall area in the image frame. At this time, the processor 140 may determine a first boundary area between the ceiling area and the wall area and a second boundary area between the floor area and the wall area in the image frame.

At this time, the processor 140 may provide a message notifying the user if the captured image frame does not include all of the wall area, ceiling area, and floor area, and may notify the user that the wall area, ceiling area, and floor area are included in the image frame. We can provide a guide to ensure that everything is included and captured. For example, the processor 140 may display a GUI including text such as “Please take a photo to include the ceiling and floor” through the display 120.

Additionally, the processor 140 may determine the extent to which the position of the wall area changes in a plurality of image frames acquired by the camera 130 while the electronic device 100 moves. At this time, it is assumed that the electronic device 100 rotates within an angle within a preset range.

Here, the processor 140 acquires the angle at which the electronic device 100 is rotated by the sensor 110, calculates the distance that the boundary area obtained in the image frame moves, and calculates the distance that the boundary area obtained in the image frame has moved. The degree to which the position of the wall area changes in the image frame can be determined.

And, the processor 140 determines the distance between the boundary corresponding to the boundary area and the electronic device 100 based on the degree to which the position of the wall area changes and the degree of movement of the electronic device 100 obtained by the sensor 110. can be judged.

Then, the processor 140 determines the distance between the electronic device 100 and the wall area based on the distance between the electronic device 100 and the boundary and the height of the electronic device 100 obtained by the sensor 110. can do.

Specifically, the processor 140 determines the height of the electronic device 100 relative to the floor based on the position of the electronic device 100 determined by the sensor 110, and determines the height of the electronic device 100 corresponding to the second boundary area. Depending on the distance between the border and the electronic device 100 and the height of the electronic device 100, the distance between the wall and the electronic device 100 can be determined using the triangular ratio.

Additionally, the processor 140 may control the display 120 to display an object on the wall area with a size corresponding to the determined distance.

Specifically, the processor 140 may determine the size corresponding to the distance determined based on the preset size and preset distance of the object, and control the display 120 so that the object is displayed on the wall area at the determined size. there is. Here, the preset size and preset distance are preset values or values set by the user, and can be changed by the user.

For example, when the preset distance between the wall area and the electronic device 100 is 10m, an object rendering a 100-inch TV (width 220cm Assume the case. When the distance between the wall area and the electronic device 100 is 5 m, the processor 140 can display the object on the wall area within the image frame to have a size of 440 x 250. When the distance between the wall area and the electronic device 100 is 15 m, the processor 140 can display the object on the wall area within the image frame so that the object has a size of 88 x 50. In this way, the distance between the wall area and the electronic device 100 and the size of the object matching the preset size may be pre-stored in the electronic device 100. Here, the preset size is a preset value or a value set by the user, and can be changed by the user.

Meanwhile, when a user command for selecting a location among the wall areas is received, the processor 140 may control the display 120 to display an object at the selected location.

3 to 6 are diagrams for explaining a control method of the electronic device 100 according to an embodiment of the present disclosure.

Referring to FIG. 3, the processor 140 may determine a monochromatic wall area in an image frame acquired by the camera 130 that performs photography using a single lens.

Specifically, the processor 140 may acquire an image frame 300 for the space 200 captured by the camera 130 using a single lens.

Here, the space 200 includes the wall 10, the ceiling 20, the floor 30, the boundary 40 of the wall 10 and the ceiling 20, the wall 10, and the floor 30. ) may include the boundary 50.

In addition, the image frame 300 includes a reference pixel 310, a wall area 320 corresponding to the wall 10, a ceiling area 330 corresponding to the ceiling surface 20, and a floor corresponding to the floor surface 30. A first boundary area 350 between the wall area 320 and the ceiling area 330 corresponding to the boundary area 40 of the area 340, the wall surface 10, and the ceiling surface 20, the wall surface 10, and It may include a second boundary area 360 between the wall area 320 and the floor area 340, which correspond to the boundary area 50 of the floor surface 30.

Meanwhile, the processor 140 may determine a monochromatic wall area 320 in the image frame 300 acquired by the camera 130. At this time, the image frame 300 may be a plurality of image frames acquired in real time by continuously photographing the space 200.

Specifically, in each of the plurality of image frames acquired by the camera 130, the processor 140 operates on an area including a reference pixel 310 and similar pixels having a color similarity greater than or equal to a preset threshold value with the reference pixel 310. can be decided. At this time, the reference pixel 310 may be set to a pixel located at the center of the image frame 300.

Then, the processor 140 compares the difference in size of the area including the reference pixel 310 and similar pixels between a plurality of image frames, and when the size difference is within a preset threshold range, the reference pixel 310 and similar pixel The area containing the pixels can be determined as the wall area 320.

Thereafter, the processor 140 determines the boundary area between the wall area and areas other than the wall area in the image frame, and determines the boundary area corresponding to the boundary area based on the movement of the electronic device 100 obtained by the sensor 110. and the distance between the electronic devices 100 may be determined.

In this case, referring to (a) of FIG. 4, the processor 140 can determine an area other than the wall area in the image frame 300 based on the wall area and determine the boundary area between the wall area and the area. there is.

To this end, the processor 140 determines a pattern area including a specific pattern in an area other than the wall area, and determines the pattern area based on the movement of the electronic device 100 acquired by the sensor 110 in each of the plurality of image frames. A set of 3D points (410, 420) can be obtained from the area.

Meanwhile, the processor 140 may determine a pattern area containing a specific pattern only for areas with a preset threshold range among areas other than the wall area. For example, the processor 140 may not determine the pattern area for the area outside the preset critical range 450 even if the pattern 460 exists in the area outside the preset critical range 450 . Accordingly, the speed and accuracy with which the processor 140 determines the pattern area can be improved.

Additionally, the processor 140 may determine the straight lines 430 and 440 corresponding to the sets of 3D points 410 and 420 as boundary areas.

Referring to (a) and (b) of FIGS. 4, the processor 140 determines the boundary between the electronic device 100 and the boundary corresponding to the boundary area based on the movement of the electronic device 100 obtained by the sensor 110. The distance can be judged.

For example, when the camera 130 of the electronic device 100 is facing the wall as shown in (a) of FIG. 4 and the camera 130 is rotated toward the floor as shown in (b) of FIG. 4 , the electronic device 100 may obtain the degree of movement (eg, rotation angle, etc.) of the electronic device 100 using the sensor 110 .

At this time, the electronic device 100 compares the center positions of the first border areas 430 and 431 with the center positions of the second border areas 440 and 441 in the plurality of image frames, respectively, to determine the first border areas 430 and 441. The degree to which the position of the wall area changes can be determined according to the degree to which the center position of 431 and the center position of the second border areas 440 and 441 change.

And, the electronic device 100 determines the distance between the boundary corresponding to the boundary area and the electronic device 100 based on the degree to which the position of the wall area changes and the degree of movement of the electronic device 100 obtained by the sensor 110. Distance can be judged.

For this purpose, information about the distance between the electronic device 100 and the boundary corresponding to the boundary area that matches the angle at which the electronic device 100 has rotated and the distance the boundary area has moved is pre-stored in the electronic device 100. There may be.

In this case, the processor 140 obtains information matching the angle at which the electronic device 100 was rotated and the distance the boundary area moved in the image frame obtained by the sensor 110 from among the previously stored distance information. , the distance between the electronic device 100 and the boundary can be determined.

For example, the electronic device 100 stores a case where the distance between the electronic device 100 and the boundary is 5 m when the angle at which the electronic device 100 is rotated is 10 degrees and the distance the boundary area moves is 10. Assume.

At this time, if the angle at which the electronic device 100 is rotated is 10 degrees and the distance the boundary area moves is 10, the processor 140 controls the electronic device 100 and the electronic device 100 according to the rotated angle and the distance the border area moves. It can be determined that the distance between boundaries is 5m.

Referring to FIG. 5, the processor 140 controls the electronic device 100 and the wall 10 based on the distance between the electronic device 100 and the boundary and the height of the electronic device 100 determined by the sensor 110. ) can determine the distance between them.

Specifically, the processor 140 may determine the distance d4 between the electronic device 100 and the wall 10 using Equation 1 below.

Here, the distance between the electronic device 100 and the second boundary 50 is d1, and the distance between the electronic device 100 and the floor is d3.

At this time, the distance d5 between the electronic device 100 and the ceiling is equal to Equation 2 below.

Here, the distance between the electronic device 100 and the first boundary 40 is d2.

Accordingly, the processor 140 calculates the distance between the electronic device 100 and the wall 10 and the height of the wall 10 (i.e., the distance between the floor and the ceiling) in real time in an image frame captured by a single lens. It can be judged as follows.

Referring to FIG. 6, the processor 140 controls the display 120 to display the object 610 for augmented reality in the wall area 320 with a size corresponding to the distance between the wall and the electronic device 100. You can.

Specifically, the processor 140 may determine a size corresponding to the distance between the wall and the electronic device 100 based on the preset size and preset distance of the object 610. Additionally, the processor 140 may control the display 120 so that the object 610 is displayed on the wall area 320 at the determined size.

To this end, information about the distance between the electronic device 100 and the wall area 320 and the size at which the object 610 matching the preset size of the object 610 will be displayed on the wall area 320 is stored in the electronic device ( 100).

The processor 140 obtains information about the distance between the wall area 320 and the electronic device 100 and the size matching the preset size of the object 610, so that the object 610 can be moved on the wall area 320. You can determine the size to be displayed.

For example, when the distance between the electronic device 100 and the wall area 320 is 10 m, the object 610 rendering a 100-inch TV (width 220 cm Assume that it is displayed in a preset size.

As shown in (a) of FIG. 6, when the distance between the electronic device 100 and the wall area 320 is 5 m, the processor 140 calculates the distance between the electronic device 100 and the wall area 320 and the object ( By obtaining a size of 440

As shown in (b) of FIG. 6, when the distance between the electronic device 100 and the wall area 320 is 15 m, the processor 140 calculates the distance between the electronic device 100 and the wall area 320 and the object ( By obtaining a size of 88

Meanwhile, the processor 140 may determine the size of the object 610 according to the height of the wall surface 10 and the height of the wall area 320 (or the distance between the first boundary and the second boundary).

According to an embodiment of the present disclosure, the electronic device 100 is configured to determine the distance between the electronic device 100 and the wall 10, the height of the electronic device 100, and the wall surface 10 in an image frame captured by a single lens. ) can be determined in real time, and accordingly, the electronic device 100 can provide an image combining objects for augmented reality on a monochromatic wall in real time.

FIG. 7 is a diagram for explaining an electronic device 100 according to another embodiment of the present disclosure.

As described above, the electronic device 100 may be implemented as various types of devices.

For example, as shown in (a) of FIG. 7, the electronic device 100 may be implemented as a wearable device that a user can wear. Here, wearable devices may be accessory-type (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD)), fabric- or clothing-integrated (e.g., electronic clothing), etc. ), body-attached, or bioimplantable circuits, etc. can be implemented as various types of devices.

Meanwhile, in the above-mentioned example, the electronic device 100 determines the distance between the electronic device 100 and the wall, and accordingly displays the image of the object for augmented reality synthesized in the wall area on the display 120. Although described, this is only an example, and as shown in (b) of FIG. 7, at least part of the functions performed by the electronic device 100 may be performed by the external electronic device 700. That is, it can be implemented through a system including the electronic device 100 and at least one external electronic device. At this time, the external electronic device 700 is shown as a TV in (b) of FIG. 7, but is not limited to this and may be implemented as various electronic devices such as a server.

First, the electronic device 100 may transmit an image captured of the wall surface and information about the location (or height) of the electronic device 100 determined by the sensor 110 to the external electronic device 700.

And, the external electronic device 700 uses the information received from the electronic device 100 (an image of the wall surface and information about the location (or height) of the electronic device 100) to connect the wall surface and the electronic device 100. You can judge the distance between them.

And, as an example, the external electronic device 700 may determine the size of the object corresponding to the determined distance and display an image composed of the object on the wall area through the display of the external electronic device 700. .

As another example, the external electronic device 700 may transmit information about the determined distance between the wall surface and the electronic device 100 to the electronic device 100. In this case, the electronic device 100 displays an object in the wall area of the image with a size corresponding to the information about the distance between the wall and the electronic device 100 received from the external electronic device 700. It can be displayed through .

As another example, the external electronic device 700 determines the distance between the wall and the electronic device 100 using information received from the electronic device 100, determines the size of the object according to the distance, and augments An image composed of real-world objects may be created, and the generated image may be transmitted to the electronic device 100. In this way, the external electronic device 700 performs a calculation operation such as determining the distance using the information received from the electronic device 100, and sends the data obtained as a result of the calculation operation to the electronic device 100. Can be transmitted. In this case, the electronic device 100 may display the image received from the external electronic device 700 through the display of the electronic device 100.

In these cases, the electronic device 100 and the external electronic device 700 use various types of data communication methods (e.g., Wi-Fi, Bluetooth, LAN, etc.) and/or various types of voice and/or video signal communication methods (e.g., Communication can be performed using DVI, HDMI, Thunderbolt, etc.).

Meanwhile, according to various embodiments of the present disclosure, the electronic device 100 determines the distance between the electronic device 100 and the wall as a result of performing voice recognition, and displays an image in which the object is synthesized in the wall area (120). ) can be displayed through. Hereinafter, the operation of performing voice recognition will be described excluding redundant content.

First, the processor 140 of the electronic device 100 is a microphone (not shown) provided inside the electronic device 100 or an external microphone (not shown) connected by wire through an input/output port (not shown) of the electronic device 100. A voice signal can be obtained by converting the user's voice, which is an analog signal, into a digital signal (not shown).

However, this is only an example, and the processor 140 of the electronic device 100 may obtain a voice signal from an external electronic device (not shown).

At this time, the electronic device 100 and the external electronic device may each include a communication unit (not shown) and may transmit and receive data through the communication unit. Here, the communication unit includes at least one of a Wi-Fi (Wi-Fi or Wi-Fi DIRECT) communication module, a Bluetooth module, an infrared communication (IrDA, infrared data association) module, an NFC (Near Field Communication) module, and a communication module. It can be done, and each module can be implemented separately or each module can be integrated.

Specifically, the external electronic device acquires a voice signal by converting the user's voice, which is an analog signal, into a digital signal using a microphone provided inside the external electronic device or an external microphone, and converts the voice signal obtained through the above-described communication unit into an electronic signal. It can be transmitted to device 100. At this time, the external electronic device may be implemented as a device such as a smartphone or remote control.

For example, when the external electronic device is implemented as a smartphone, an application capable of controlling the electronic device 100 may be installed or stored in the external electronic device, and the electronic device 100 such as a TV or set-top box can be controlled through the application. ) can be implemented to control. Furthermore, the external electronic device may directly perform voice recognition on the acquired voice signal, or may transmit the results of voice recognition to the electronic device 100 through a server.

As another example, when the external electronic device is implemented as a remote control, the external electronic device may have an input unit (e.g., button, joystick, etc.) capable of controlling the electronic device 100, and may operate the TV or It may be implemented to control an electronic device 100 such as a set-top box.

Next, the processor 140 of the electronic device 100 may perform voice recognition on the acquired voice signal and control the electronic device 100 to perform an operation corresponding to the result of voice recognition.

To this end, the processor 140 of the electronic device 100 may transmit the acquired voice signal to a voice recognition server (not shown).

Additionally, when a user's voice is received, the voice recognition server can convert the received user's voice into text using the STT algorithm. The voice recognition server can obtain response information that allows the user to perform the requested action based on the converted text. At this time, the voice recognition server can be implemented as a main server that performs both the STT (Speech to Text) function and the function of obtaining response information.

Additionally, the voice recognition server may transmit response information to the electronic device 100 that causes the electronic device 100 to perform the operation requested by the user.

However, this is only an example, and the voice recognition server may be implemented as a system including a first server performing the STT function and a second server performing the function of obtaining response information. In this case, the first server converts the received user voice into text and transmits it to the second server, and the second server obtains response information based on the received text and transmits it to the electronic device 100. Meanwhile, the first server may transmit the converted text to the electronic device 100, and the electronic device 100 may obtain response information based on the received text.

Additionally, the processor 140 of the electronic device 100 may control the electronic device 100 to perform an operation requested by the user according to the response information.

Meanwhile, in another embodiment, the processor 140 of the electronic device 100 performs a Speech to Text (STT) function to convert text and obtain response information accordingly. You can also perform voice recognition directly.

FIG. 8 is a block diagram illustrating in detail the configuration of the electronic device 100 according to an embodiment of the present disclosure.

Referring to FIG. 8, the electronic device 100 includes one of an input unit 150, a speaker 160, and a communication unit 170 in addition to a sensor 110, a display 120, a camera 130, and a processor 140. It can be included.

The processor 140 may include RAM 141, ROM 142, graphics processing unit 143, main CPU 144, first to n interfaces (145-1 to 145-n), and bus 146. there is. At this time, RAM 141, ROM 142, graphics processing unit 143, main CPU 144, first to n interfaces (145-1 to 145-n), etc. may be connected to each other through bus 146. .

The memory 150 can store various programs and data necessary for the operation of the electronic device 100.

The memory 150 may be implemented as non-volatile memory, volatile memory, flash-memory, hard disk drive (HDD), or solid state drive (SSD). The memory 150 is accessed by the processor 140, and data read/write/modify/delete/update, etc. can be performed by the processor 140. The term memory in the present disclosure refers to memory 150, RAM 141 in processor 140, ROM 142, or memory card (not shown) mounted on electronic device 100 (e.g., micro SD card, memory stick, etc.).

The input unit 160 may receive various user inputs and transmit them to the processor 140. The input unit 160 may include, for example, a touch panel, a (digital) pen sensor, or a key. The touch panel may use at least one of, for example, a capacitive type, a resistive type, an infrared type, or an ultrasonic type. Additionally, the touch panel may further include a control circuit. The touch panel may further include a tactile layer to provide a tactile response to the user. The (digital) pen sensor may be, for example, part of a touch panel or may include a separate recognition sheet. Keys may include, for example, physical buttons, optical keys, or keypads. Alternatively, the input unit 160 may be connected wired or wirelessly from an external device (not shown) such as a keyboard, mouse, etc. to receive user input.

The input/output port 170 allows the electronic device 100 to transmit and/or receive signals and/or data for images and/or voices with an external device (not shown). (not shown) is connected by wire. At this time, the input/output port 170 may include a module that processes transmitted or received signals.

To this end, the input/output port 170 may be implemented as a wired port such as an HDMI port, display port, RGB port, DVI (Digital Visual Interface) port, Thunderbolt, and component port.

As an example, so that the electronic device 100 can output the image and/or voice, the electronic device 100 receives a signal for the image and/or voice from an external device (not shown) through the input/output port 170. In another example, the electronic device 100 sends a signal for a specific image and/or voice to an external device through the input/output port 170 so that an external device (not shown) can output the image and/or voice. It can be sent to the device.

In this way, signals for images and/or voices can be transmitted in one direction through the input/output port 170. However, this is only an example, and of course, signals for images and/or voices can be transmitted in both directions through the input/output port 170.

Additionally, the input/output port 170 may include a USB port (2.0, USB 3.0, USB C, etc.), a Secure Digital (SD) card port, and a Secure Digital (Micro SD) card port.

Meanwhile, the electronic device 100 may include a microphone (not shown) and a speaker (not shown). A microphone (not shown) can directly acquire a voice signal from external sounds. A speaker (not shown) can directly output various notification sounds or voice messages as well as various audio data on which various processing tasks such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown).

The communication unit 180 can communicate with various types of external devices according to various types of communication methods. The communication unit 180 may include at least one of a Bluetooth chip (not shown), a Wi-Fi chip (not shown), a wireless communication chip (not shown), and an NFC chip (not shown).

Here, the communication unit 180 is controlled by the processor 140 and can communicate with various external devices (not shown) such as other electronic devices and servers.

The communication unit 180 can transmit and receive data with an external device (not shown) through wireless communication. In this case, the communication unit 180 includes a Wi-Fi (Wi-Fi or Wi-FiDIRECT) communication module, a Bluetooth module, and an infrared ray. Communication (IrDA, infrared data association) module, NFC (Near Field Communication) module, Zigbee module, cellular communication module, 3G (3rd generation) mobile communication module, 4G (4th generation) communication module, 4th generation LTE (Long It may include at least one of a Term Evolution) communication module and a 5G (5th generation) communication module.

Meanwhile, the communication unit 180 can transmit and receive data with an external device (not shown) through a wired communication method through an input/output port. In this case, the communication unit 180 may include a wired LAN module, an Ethernet module, etc.

FIG. 9 is a flowchart for explaining a control method of the electronic device 100 according to an embodiment of the present disclosure.

Referring to FIG. 9, first, the boundary area between the wall area and areas other than the wall area can be determined in the image frame acquired by the camera 130 (S910). At this time, the camera can capture images of a monochromatic wall using a single lens, and obtain an image frame that includes a wall area corresponding to the monochromatic wall as a result of the imaging.

Here, in each of the plurality of image frames acquired by the camera 130, a reference pixel existing at a preset position and a similar pixel having a color similarity greater than a preset threshold are determined, and the reference pixel and If the difference in size of the area including the similar pixel is within a preset threshold range, the area including the reference pixel and the similar pixel may be determined to be a wall area.

In addition, it is possible to determine a pattern area containing a specific pattern in an area other than the wall area, obtain a set of 3D points based on the pattern area, and determine the boundary area according to a straight line corresponding to the set of 3D points. there is.

Here, the area other than the wall area includes a ceiling area that exists above the wall area in the image frame and a floor area that exists below the wall area, and in this case, the first boundary between the ceiling area and the wall area in the image frame. A second boundary area between the area and the floor area and the wall area may be determined.

Next, based on the location of the electronic device determined by the sensor 110 and the determined boundary area, the distance between the wall surface and the electronic device 100 may be determined (S920).

To this end, while the electronic device 100 is moving, the change in the position of the electronic device 110 and the change in the position of the wall area are determined in the plurality of image frames acquired by the camera 130, and the change in the position of the wall area is determined. Depending on the degree of change in position and the degree of change in the position of the electronic device 100 determined by the sensor 110, the distance between the boundary corresponding to the boundary area and the electronic device 100 may be determined.

And, based on the distance between the boundary and the electronic device 100 corresponding to the boundary area and the position of the electronic device 100 determined by the sensor 110, the wall surface and the electronic device 100 corresponding to the wall area and You can judge the distance between them.

Here, the height of the electronic device 100 relative to the floor is determined based on the position of the electronic device 100 determined by the sensor 110, and the boundary and electronic device corresponding to the second boundary area are determined. The distance between the electronic device 100 and the wall area may be determined based on the distance between them 100 and the height of the electronic device 100.

Next, the object can be displayed on the wall area with a size corresponding to the determined distance (S930).

Here, the size at which an object with a preset size will be displayed on the wall area may be determined based on the distance between the wall area and the electronic device 100, and the object may be displayed on the wall area at the determined size.

Meanwhile, when an input for selecting a wall area is received, an object may be displayed at the selected location in the wall area based on the received input.

According to various embodiments of the present disclosure as described above, an electronic device capable of providing an image generating a virtual object on a monochromatic wall of an image captured by a single lens and a control method thereof can be provided.

In addition, the accuracy of distance measurement using a single lens can be improved, and compared to the case of a dual lens, the product can be miniaturized and simplified, and costs can be reduced.

The term “unit” or “module” used in this disclosure includes a unit comprised of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. . A “part” or “module” may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. For example, a module may be comprised of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media that can be read by a machine (e.g., a computer). The device calls the stored instructions from the storage medium. and, as a device capable of operating according to the called instruction, it may include an electronic device (e.g., electronic device 100) according to the disclosed embodiments. When the instruction is executed by a processor, the processor may directly, or Functions corresponding to the instructions can be performed using other components under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. Device-readable storage media are non-transitory. It can be provided in the form of a (non-transitory) storage medium. Here, 'non-transitory' only means that the storage medium does not contain signals and is tangible, and that the data is not semi-permanent or temporary in the storage medium. There is no distinction between being stored as .

Methods according to various embodiments may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed on a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or online through an application store (e.g. Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or created temporarily in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

Each component (e.g., module or program) according to various embodiments may be composed of a single or plural entity, and some of the above-described sub-components may be omitted, or other sub-components may be variously used. Further examples may be included. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added. You can.

100: electronic device

Claims (18)

In electronic devices,
sensor;
display;
camera; and
Obtaining the illuminance value through the sensor,
If the illuminance value is greater than or equal to a preset value, determining a similar pixel having a color similarity greater than or equal to a preset threshold value with a reference pixel located at the center in each of the plurality of image frames acquired by the camera;
If the size difference of the area including the reference pixel and the similar pixel between the plurality of image frames is within a preset threshold range, determining the area including the reference pixel and the similar pixel as a wall area;
Determine a boundary area between the wall area and an area other than the wall area,
Determining a distance between the electronic device and a wall corresponding to the wall area based on the location of the electronic device determined by the sensor and the determined border area,
It includes a processor that controls the display so that an object is displayed on the wall area with a size corresponding to the determined distance,
Areas other than the wall area,
Each of the plurality of image frames includes a ceiling area above the wall area and a floor area below the wall area,
The processor,
Determine a first boundary area between the ceiling area and the wall area and a second boundary area between the floor area and the wall area in each of the plurality of image frames,
Determining the height of the electronic device relative to the floor using the sensor,
An electronic device that determines a distance between a wall surface corresponding to the wall area and the electronic device based on a distance between the electronic device and a second boundary corresponding to the second boundary area, and a height of the electronic device. delete According to paragraph 1,
The processor,
Determine a pattern area containing a specific pattern in an area other than the wall area,
Obtaining a set of three-dimensional points in the pattern area based on the movement of the electronic device determined by the sensor,
An electronic device that determines the boundary area according to a straight line corresponding to the set of three-dimensional points. According to paragraph 1,
The processor,
Determine the extent to which the position of the wall area changes in the plurality of image frames acquired by the camera while the electronic device moves,
An electronic device that determines a distance between a boundary corresponding to the boundary area and the electronic device based on the degree to which the position of the wall area changes and the degree of movement of the electronic device determined by the sensor. delete delete delete According to paragraph 1,
The processor,
Determine a size corresponding to the determined distance based on the preset size and preset distance of the object,
An electronic device that controls the display so that the object is displayed in the wall area at the determined size. According to paragraph 1,
The processor,
An electronic device that controls the display to display the object at the selected location when a user command for selecting a location among the wall areas is received. In a method of controlling an electronic device,
Obtaining an illuminance value through a sensor;
If the illuminance value is greater than or equal to a preset value, determining a similar pixel having color similarity greater than or equal to a preset threshold value with a reference pixel located at the center in each of the plurality of image frames acquired by the camera;
If a size difference between the reference pixels and the similar pixels between the plurality of image frames is within a preset threshold range, determining the area including the reference pixels and the similar pixels as a wall area;
determining a boundary area between the wall area and an area other than the wall area;
determining a distance between the electronic device and a wall corresponding to the wall area based on the location of the electronic device determined by the sensor and the determined boundary area; and
It includes: displaying an object on the wall area with a size corresponding to the determined distance,
Areas other than the wall area,
Each of the plurality of image frames includes a ceiling area above the wall area and a floor area below the wall area,
The step of determining the boundary area is,
Comprising: determining a first boundary area between the ceiling area and the wall area and a second boundary area between the floor area and the wall area in each of the plurality of image frames,
The step of determining the distance between the wall and the electronic device includes:
determining the height of the electronic device relative to the floor using the sensor; and
Determining the distance between the electronic device and a wall surface corresponding to the wall area based on the distance between the second border corresponding to the second boundary area and the electronic device, and the height of the electronic device; comprising; , control method. delete According to clause 10,
The step of determining the boundary area is,
Determine a pattern area containing a specific pattern in an area other than the wall area,
Obtaining a set of three-dimensional points in the pattern area based on the movement of the electronic device determined by the sensor,
A control method for determining the boundary area according to a straight line corresponding to the set of three-dimensional points. According to clause 10,
The step of determining the distance between the wall and the electronic device includes:
Determine the extent to which the position of the wall area changes in the plurality of image frames acquired by the camera while the electronic device moves,
Control comprising determining a distance between a boundary corresponding to the boundary area and the electronic device based on the degree to which the position of the wall area changes and the degree of movement of the electronic device determined by the sensor. method. delete delete delete According to clause 10,
The steps indicated above are:
Determine a size corresponding to the determined distance based on the preset size and preset distance of the object,
A control method for displaying the object at the determined size on the wall area. According to clause 10,
The steps indicated above are:
When an input for selecting a location among the wall areas is received, displaying the object at the selected location.

Images