KR20240045049A - An electronic apparatus and a method thereof - Google Patents

Abstract

It includes a projection unit, a memory for storing one or more instructions, and one or more processors for executing one or more instructions stored in the memory, wherein the one or more processors execute one or more instructions to create a projection space corresponding to the projection space from modeling information about the surrounding space. Obtain modeling information, correct the shape of the standard object content based on the modeling information corresponding to the projection space, obtain projection object content, and determine the operation of the projection object content based on the modeling information corresponding to the projection space. An electronic device is disclosed that projects projection content including projection object content that performs a determined operation into a projection space through a projection unit.

Description

An electronic apparatus and a method thereof}

Various disclosed embodiments relate to an electronic device and a method of operating the same, and more specifically, to an electronic device that projects content and a method of operating the same.

The technology that overlaps virtual images on the real world is called augmented reality. Augmented reality is different from virtual reality, where users interact with the surrounding environment in a virtual space, in that it displays virtual images overlaid on the physical real world.

With the development of optical technology, various types of projectors are being developed. A projector is an electronic device that projects images by projecting light onto a certain space or plane, and can implement augmented reality technology by projecting virtual images into the real world.

An electronic device according to an embodiment may include a projection unit, a memory that stores one or more instructions, and one or more processors that execute one or more instructions stored in the memory.

In an embodiment, the one or more processors may obtain modeling information corresponding to the projection space from modeling information about the surrounding space by executing the one or more instructions. ,

In an embodiment, the one or more processors may acquire projection object content by executing the one or more instructions, thereby correcting the shape of standard object content based on modeling information corresponding to the projection space.

In an embodiment, the one or more processors may determine an operation of the projection object content based on modeling information corresponding to the projection space by executing the one or more instructions.

In an embodiment, the one or more processors may project projection content including the projection object content that performs the determined operation into the projection space through the projection unit by executing the one or more instructions.

A method of operating an electronic device according to an embodiment may include obtaining modeling information corresponding to the projection space from modeling information about the surrounding space.

A method of operating an electronic device according to an embodiment may include obtaining projection object content by correcting the shape of standard object content based on modeling information corresponding to the projection space.

A method of operating an electronic device according to an embodiment may include determining an operation of the projection object content based on modeling information corresponding to the projection space.

A method of operating an electronic device according to an embodiment may include projecting projection content including the projection object content that performs the determined operation into the projection space.

A recording medium according to an embodiment is a computer-readable program on which a program capable of executing a method of operating an electronic device, including the step of acquiring modeling information corresponding to the projection space from modeling information about the surrounding space, is recorded. It may be a recording medium.

In an embodiment, the recording medium may perform a method of operating an electronic device by a computer, including obtaining projection object content by correcting the shape of standard object content based on modeling information corresponding to the projection space. It may be a computer-readable recording medium on which a program is recorded.

In an embodiment, the recording medium records a program capable of performing, by a computer, a method of operating an electronic device, including determining an operation of the projection object content based on modeling information corresponding to the projection space. It may be a computer-readable recording medium.

In an embodiment, the recording medium includes a program that can perform a method of operating an electronic device by a computer, including the step of projecting projection content including the projection object content that performs the determined operation into the projection space. It may be a recorded, computer-readable recording medium.

FIG. 1 is a diagram illustrating how an electronic device projects virtual content onto the real world, according to an embodiment.
Figure 2 is an internal block diagram of the electronic device 100 according to an embodiment.
Figure 3 is an internal block diagram of the electronic device 100 according to an embodiment.
FIG. 4 is a diagram illustrating an operation module of the processor 101 inside the electronic device 100 according to an embodiment.
FIG. 5 is a diagram for explaining rotation of the electronic device 100 according to an embodiment.
FIG. 6 is a diagram to explain how the electronic device 100 acquires modeling information about its surroundings, according to an embodiment.
FIG. 7 is a diagram for explaining how the electronic device 100 identifies a projection direction, according to an embodiment.
FIG. 8 is a diagram for explaining how the electronic device 100 corrects the shape of object content based on modeling information corresponding to the projection space, according to an embodiment.
FIG. 9 is a diagram for explaining how the electronic device 100 corrects the shape of object content based on modeling information corresponding to the projection space, according to an embodiment.
FIG. 10 is a diagram illustrating how the electronic device 100 corrects the shape of standard object content based on modeling information corresponding to the projection space, according to an embodiment.
FIG. 11 is a diagram explaining how the electronic device 100 determines the movement of object content according to a projection space, according to an embodiment.
FIG. 12 is a diagram explaining how the electronic device 100 recognizes the amount of change in the projection space and controls the operation of projection object content accordingly, according to an embodiment.
FIG. 13 is a diagram explaining how the electronic device 100 adjusts the size of background content, according to an embodiment.
Figure 14 is a flowchart illustrating a method of obtaining modeling information about surrounding space, according to an embodiment.
FIG. 15 is a flowchart illustrating a method of operating the electronic device 100 according to an embodiment.
FIG. 16 is a flowchart illustrating controlling the operation of projection object content differently based on a difference in depth value of the projection space, according to an embodiment.

In the present disclosure, the expression “at least one of a, b, or c” refers to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b and c", or variations thereof.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

The terms used in this disclosure are described as general terms currently used in consideration of the functions mentioned in this disclosure, but may mean various other terms depending on the intention or precedents of those skilled in the art, the emergence of new technologies, etc. You can. Accordingly, the terms used in this disclosure should not be interpreted only by the name of the term, but should be interpreted based on the meaning of the term and the overall content of this disclosure.

Additionally, the terms used in the present disclosure are merely used to describe specific embodiments and are not intended to limit the present disclosure.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. .

As used in this specification, particularly in the patent claims, “the” and similar indicators may refer to both the singular and the plural. Additionally, unless there is a description clearly designating the order of steps describing the method according to the present disclosure, the steps described may be performed in any suitable order. The present disclosure is not limited by the order of description of the steps described.

Phrases such as “in some embodiments” or “in one embodiment” that appear in various places in this specification do not necessarily all refer to the same embodiment.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit configurations for certain functions. Additionally, for example, functional blocks of the present disclosure may be implemented in various programming or scripting languages. Functional blocks may be implemented as algorithms running on one or more processors. Additionally, the present disclosure may employ conventional technologies for electronic environment setup, signal processing, and/or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical components.

Additionally, connection lines or connection members between components shown in the drawings merely exemplify functional connections and/or physical or circuit connections. In an actual device, connections between components may be represented by various replaceable or additional functional connections, physical connections, or circuit connections.

In addition, terms such as "... unit" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. .

Additionally, the term “user” in the specification refers to a person who uses an electronic device and may include a consumer, evaluator, viewer, administrator, or installer.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

FIG. 1 is a diagram illustrating how an electronic device projects virtual content onto the real world, according to an embodiment.

Referring to FIG. 1, the electronic device 100 may be an electronic device capable of outputting images. According to one example, the electronic device 100 may be a projector that projects an image into a projection space. The electronic device 100 may be fixed or mobile.

FIG. 1 illustrates a case where the electronic device 100 is mounted on the ceiling and projects projection content to the surroundings, according to an example. For example, the electronic device 100 may be connected to the ceiling through a connection member.

In an embodiment, the electronic device 100 may be rotatable. For example, the electronic device 100 may be rotated by being connected to a connection member in a rotatable form.

In an embodiment, the electronic device 100 may include a projection unit. In an embodiment, the projection unit may project projection content.

In FIG. 1, assuming that the electronic device 100 is mounted on the ceiling and projects projection content onto the floor, the floor becomes the projection space. When the projection direction and the projection space form a right angle, the projection direction of light projected toward the center of the projection space or the direction opposite to the projection of light can be defined as the z-axis. Additionally, two mutually perpendicular axes on a plane perpendicular to the z-axis, that is, a plane parallel to the floor, can be defined as the x-axis and y-axis, respectively.

In an embodiment, the electronic device 100 may rotate about each of the x-axis, y-axis, and z-axis. For example, the rotation angles at which the electronic device 100 rotates about the x-axis, y-axis, and z-axis are respectively a roll angle (ψ), a pitch angle (θ), and a yaw angle (ψ). It may be referred to as . In an embodiment, the rotation angle at which the electronic device 100 rotates about the x-axis, y-axis, and z-axis may vary depending on the structure of the electronic device 100 or its connected state to a fixed surface. For example, in FIG. 1, the electronic device 100 can rotate by 180 degrees along the y-axis and by 360 degrees along the x-axis and z-axis, but is not limited thereto.

Alternatively, the entire electronic device 100 may not rotate, but only some components included in the electronic device 100 may rotate. For example, the electronic device 100 may include a rotatable member. Components such as a projection unit, a camera sensor, and a depth sensor are disposed on the rotating member, and as the rotating member rotates, components such as the projection unit, a camera sensor, and a depth sensor may also rotate.

In an embodiment, the electronic device 100 may include an image sensor and a depth sensor. In an embodiment, the image sensor and the depth sensor may be implemented as separate sensors, but are not limited to this and may also be implemented as a single sensor. For example, the image sensor may be a depth sensor with a depth function.

In an embodiment, the image sensor and the depth sensor may acquire information about the surrounding space while rotating according to the rotation of the electronic device 100 or according to the rotation of the rotating member.

In embodiments, the image sensor may include a camera. In an embodiment, the image sensor may acquire an image of the surrounding space by photographing the surrounding space while rotating.

In an embodiment, the depth sensor may acquire depth information of the surrounding space while rotating.

In an embodiment, the electronic device 100 may obtain modeling information about the surrounding space based on the surrounding space image and surrounding space depth information. When the electronic device 100 rotates and moves, the rotation angle for each axis changes, and the electronic device 100 generates modeling information at that point according to the surrounding space image acquired at that angle and the depth information of the surrounding space. can do.

In an embodiment, modeling may mean creating a virtual space that resembles a surrounding three-dimensional space. In an embodiment, the modeling information may include color information, luminance information, and depth information according to the rotation angle for each axis of at least one axis of the electronic device 100 or the rotating member.

In an embodiment, the electronic device 100 may identify a projection space. In an embodiment, the projection space may mean a space or area where projection content is projected.

In an embodiment, the electronic device 100 may identify the projection space based on the user direction. In an embodiment, the electronic device 100 may photograph a user using an image sensor. In an embodiment, the electronic device 100 may identify user characteristic points from a user image obtained by photographing the user. In an embodiment, a user characteristic point may be information for identifying a part of the user's body.

In an embodiment, the electronic device 100 may identify a point on the user's body based on user characteristic points obtained from the user image. For example, the electronic device 100 may identify predefined user characteristic points, such as the back of the user's head, ears, neck, shoulders, forehead, nose, etc., in the user image.

In an embodiment, the electronic device 100 may connect identified user feature points to generate a three-dimensional figure corresponding to a part of the user's body. For example, the electronic device 100 may connect feature points to create a three-dimensional figure that fits the user's head.

In an embodiment, the electronic device 100 may identify the location of the user's face based on a three-dimensional figure. In an embodiment, the electronic device 100 may measure the degree of tilt of a three-dimensional figure and identify the direction in which the user's face is facing according to the degree of tilt.

In an embodiment, the electronic device 100 may identify the projection space based on the direction the user's face faces. For example, when the direction in which the user's face faces is centered at the point where it meets the surrounding space, the electronic device 100 may identify an area at a predetermined distance from the center as the projection space.

In an embodiment, the electronic device 100 may obtain modeling information corresponding to the projection space from modeling information about the surrounding space. In an embodiment, the electronic device 100 may identify modeling information about a projection space located in the direction where the user's face faces, among modeling information about the surrounding space, and obtain this as modeling information corresponding to the projection space.

In an embodiment, the electronic device 100 may acquire standard object content. In an embodiment, standard object content is content created by a content production company, etc., and may have standard color, brightness, shape, size, etc. set by the production company. In an embodiment, standard object content may be produced to have a certain size, shape, color, brightness, etc. when projected into a projection space with color, brightness, spatial structure, etc. of predetermined standard values.

In embodiments, standard object content may be a virtual image that imitates an actor, movie character, animation, or animal that exists in the real world or is imaginary.

In an embodiment, the electronic device 100 may receive standard object content of various types and forms from an external server through a communication network. Alternatively, in an embodiment, the electronic device 100 may receive a user's selection of one or more standard object contents previously stored in the internal memory. Alternatively, in an embodiment, if the electronic device 100 is capable of generating standard object content, the electronic device 100 may newly generate standard object content for people, animals, objects, etc. included in the image input by the user. It may be possible.

In an embodiment, the electronic device 100 may correct standard object content based on modeling information.

Standard object content is produced to have a certain size, shape, color, brightness, etc. when projected into a projection space with predetermined standard values of color, brightness, spatial structure, etc., so the color, brightness, spatial structure, etc. of the projection space are standard. If it does not have a value, the size, shape, color, or brightness of the standard object content projected into such projection space will also have a different form from the standard object content.

In an embodiment, the electronic device 100 may correct the color, brightness, shape, size, etc. of standard object content based on modeling information corresponding to the projection space. In an embodiment, the electronic device 100 corrects the color, brightness, shape, size, etc. of standard object content based on modeling information corresponding to the projection space, even if the projection space is not a space with a predetermined standard value. Projected object content can have the size, color, brightness, shape, etc. of standard object content.

When modeling information corresponding to the projection space has different colors, brightness, etc. for each region, in an embodiment, the electronic device 100 displays the color, brightness, shape, size, etc. of partial regions forming the projected object content in the projection space. Each can be corrected based on the corresponding modeling information. For example, the electronic device 100 may correct the color, brightness, shape, size, etc. for each pixel of each of the plurality of areas constituting the standard object content.

In an embodiment, the electronic device 100 may adjust the overall size of standard object content according to the distance between the user and the projection space.

If standard object content appears to be the same size when the user is close to the projection space and when it is far away, realism is reduced.

In an embodiment, the electronic device 100 may obtain depth information up to the user and depth information up to the projection space, respectively. In an embodiment, the electronic device 100 may obtain the distance between the user and the projection space by calculating the difference between the depth information to the user and the depth information to the projection space.

In an embodiment, the electronic device 100 may adjust the overall size of standard object content based on the distance between the user and the projection space. For example, when the distance between the user and the projection space is the standard distance, the electronic device 100 displays standard object content at a standard size, and when the distance between the user and the projection space is shorter than the standard distance, the electronic device 100 displays standard object content to the user. You can correct the size of standard object content to make it appear larger than the standard size. Additionally, when the distance between the user and the projection space is longer than the standard distance, the electronic device 100 may correct the size of the standard object content so that the standard object content appears smaller than the standard size to the user.

In an embodiment, the electronic device 100 may project standard object content with corrected color, brightness, shape, size, etc. into the projection space.

In the present disclosure, content obtained by the electronic device 100 by correcting standard object content for projection in a projection space is referred to as projection object content.

The electronic device 100 may obtain projection object content by correcting standard object content based on modeling information corresponding to the projection space and project it into the projection space.

In an embodiment, the electronic device 100 may generate background content. In embodiments, the background content may be content surrounding the projection object content.

In an embodiment, the electronic device 100 may generate background content based on modeling information corresponding to the projection space. For example, the electronic device 100 may determine at least one of the color and brightness of the background content based on at least one of the color and brightness of modeling information corresponding to the projection space.

Alternatively, the electronic device 100 may determine the brightness of the background content to be the minimum value so that the projected background content is not well recognized in the projection space.

In an embodiment, the electronic device 100 may generate projection content including projection object content and background content. In an embodiment, the electronic device 100 may project the projection content 110 into a projection space.

FIG. 1 shows a case where the projection object content included in the projection content 100 is a dog. As shown in FIG. 1, when the projection object content is a virtual image that imitates an animal that exists in the real world, the projection object content may be treated as a type of virtual cyber animal. Users can use object content by interacting with cyber animals in various ways, such as raising or playing with virtual cyber animals projected in the projection space by the electronic device 100.

In an embodiment, the projected object content may be a virtual image with movement. In an embodiment, the electronic device 100 may control the movement of projection object content. In an embodiment, the electronic device 100 may control the movement of projection object content based on at least one of a preset period, a random period, and event occurrence.

In an embodiment, the electronic device 100 may control the movement of projection object content at preset intervals, for example, every 5 seconds.

In an embodiment, the electronic device 100 may control projection object content to operate in a predefined order. For example, the electronic device 100 may control the projection object content to perform a sitting motion and, when 5 seconds have elapsed, control the projection object content to perform a walking motion. The electronic device 100 may control the projected object content to perform the motion of sniffing the floor after another 5 seconds.

Alternatively, in an embodiment, the electronic device 100 may control projection object content to perform random operations. For example, after controlling the projection object content to perform a sitting motion, the electronic device 100 controls the projection object content to perform a sitting motion, and then after a certain time, for example, 3 seconds, performs a random motion different from before, for example, the projection object content sniffing the floor. , and when a certain time, for example, 7 seconds, has elapsed, the projection object content can be controlled to perform another arbitrary action, for example, a walking motion.

In an embodiment, the electronic device 100 may control the movement of projection object content whenever an event occurs. In an embodiment, event occurrence may include a case where the direction of the user's face changes by more than a threshold and the position of the projection space also changes by more than a threshold.

In an embodiment, when the direction of the user's face changes by more than a threshold, the electronic device 100 may re-determine the projection space in the direction the user's face faces. In an embodiment, the electronic device 100 may re-determine the movement of projection object content when the projection space changes.

In embodiments, generating an event may include receiving a projection object content operation control signal from a user. In an embodiment, a user may input a projection object content operation control signal to the electronic device 100 by making a specific gesture or uttering a specific word.

In an embodiment, when the user performs a predefined specific action or utters a predefined specific word, the electronic device 100 receives this as a projection object content operation control signal, and based on this, controls the projection object content. Movement can be controlled.

In an embodiment, when the electronic device 100 receives standard object content from an external server, it may also receive operation information indicating the operation performed by the standard object content. Additionally, when standard object content is stored in the internal memory of the electronic device 100, operation information for each standard object content may also be stored in the memory.

In an embodiment, the motion information may be information indicating various characteristics or motions of standard object content. For example, operation information may be information indicating operations normally performed by the corresponding object depending on the object type.

In an embodiment, a content provider may generate operation information based on an object type, various operations that an object belonging to that type can generally perform. For example, if the object is a dog, the content provider may provide various types of movements performed by a typical dog, such as wagging its tail, breathing with its tongue out, crouching, walking or running, and lying down with its belly exposed. etc. can be generated as motion information.

Alternatively, the content provider may also generate operations that the corresponding object cannot perform as operation information. For example, a content provider can generate the motion of an object crawling on a wall like a lizard as motion information even though the object is a dog.

Alternatively, the motion information may be information representing the object's unique habits, etc. For example, a content provider can generate the habitual movement of an object tilting its head as motion information.

When the user selects one of the standard object contents already stored in the electronic device 100 or creates new standard object content by modifying the selected standard object content, the user determines what operation the selected or created standard object content performs. You can also select whether to perform a specific operation, or have new operation information created by performing a specific operation.

In an embodiment, the electronic device 100 may control the movement of standard object content according to motion information.

In an embodiment, the electronic device 100 may control projection object content to move differently depending on the projection space.

In an embodiment, modeling information corresponding to the projection space may include projection space structure information. In an embodiment, the projection space structure information may include information about the arrangement or position of the structure in the projection space, and/or spatial change information according to the arrangement or position of the structure in the projection space.

In an embodiment, the electronic device 100 may determine the operation of projection object content as an operation corresponding to projection space structure information. For example, when the projection object content is near a sofa, the electronic device 100 may control the projection object content to sit on the sofa. Alternatively, when the projected object content is near the bed, the electronic device 100 may control the user to perform an action of curling up or lying down under the bed or on top of the bed.

In an embodiment, when there is a structure in the projection space, the electronic device 100 may determine the movement direction or motion of the projection object content so that the projection object content does not collide with the structure. For example, if there is a refrigerator in the projection space, the electronic device 100 may control the projection object content to walk around the refrigerator.

In an embodiment, the electronic device 100 may control the operation of projection object content differently depending on the degree of spatial change. In an embodiment, the electronic device 100 may control the operation of projection object content differently in a projection space where there is a spatial change below a threshold and a projection space where there is a spatial change that exceeds the threshold.

In an embodiment, a projection area in which there is a spatial change greater than a threshold may mean a space in which the spatial change differs by a certain amount or more along at least one of the x-axis, y-axis, and z-axis.

In an embodiment, the electronic device 100 may control the operation of projection object content as a first operation in a projection space where there is a spatial change below a threshold. For example, the electronic device 100 may control projection object content to move according to the first movement method in a general projection space.

On the other hand, in an embodiment, the electronic device 100 may determine the motion of the projection object content in a projection space where there is a spatial change exceeding the threshold as a second motion different from the first motion. Unlike the first motion, the second motion may be a specific movement method that the projected object content can perform in a space with large changes.

For example, as shown in FIG. 1, when the object content is a dog, the first motion may be a motion such as crawling, stretching, crouching, or wagging the tail. Additionally, the second motion may be an motion such as jumping up, jumping down, or jumping.

As such, according to the embodiment, the electronic device 100 may correct the shape of the projection object content based on modeling information corresponding to the projection space.

Additionally, according to an embodiment, the electronic device 100 may determine the operation of projection object content based on modeling information corresponding to the projection space. Therefore, users can experience augmented reality in which virtual images naturally overlap as if they actually exist in the real world.

Figure 2 is an internal block diagram of the electronic device 100 according to an embodiment.

Referring to FIG. 2 , the electronic device 100 may include a processor 101, a memory 103, and a projection unit 105.

The memory 103 according to an embodiment may store at least one instruction. The memory 103 may store at least one program that the processor 101 executes. Additionally, the memory 103 may store data input to or output from the electronic device 100.

The memory 103 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , and may include at least one type of storage medium among optical disks.

In an embodiment, one or more instructions for obtaining modeling information about the surrounding space may be stored in the memory 103.

In an embodiment, one or more instructions for obtaining modeling information about the surrounding space may be stored in the memory 103 based on the surrounding space image and surrounding space depth information.

In an embodiment, the memory 103 may store one or more instructions for identifying the direction in which the user's face faces based on the user's feature points and identifying a space located in the direction in which the user's face faces as a projection space.

In an embodiment, one or more instructions for obtaining modeling information corresponding to the projection space from modeling information about the surrounding space may be stored in the memory 103.

In an embodiment, memory 103 may store one or more instructions for obtaining standard object content.

In an embodiment, the memory 103 may store various types of standard object content and operation information of the standard object content.

In an embodiment, the memory 103 may store one or more instructions for recognizing an object from an input image and generating standard object content and operation information of the standard object content for the recognized object.

In an embodiment, the memory 103 may store one or more instructions for correcting projection object content based on modeling information corresponding to the projection space.

In an embodiment, the memory 103 may store one or more instructions for correcting at least one of color, brightness, shape, and size of projection object content based on modeling information corresponding to the projection space.

In an embodiment, one or more instructions for correcting the size of projection object content may be stored in the memory 103 using projection space depth information and user depth information.

In an embodiment, the memory 103 may store one or more instructions for determining an operation of the content of the projection object as an operation corresponding to the projection space structure information.

In an embodiment, memory 103 may store one or more instructions for determining an operation of projection object content as a first operation in a projection space where there is a spatial change below a threshold.

In an embodiment, memory 103 may store one or more instructions for determining the motion of projection object content as a second motion different from the first motion in a projection space where there is a spatial change exceeding a threshold.

In an embodiment, the memory 103 may store one or more instructions for determining at least one of the color and brightness of the background content based on at least one of the color and brightness of modeling information corresponding to the projection space.

In an embodiment, memory 103 may store one or more instructions for determining the brightness of background content to a minimum value.

In an embodiment, the memory 103 may store one or more instructions for projecting projection content including projection object content and background content into a projection space.

In an embodiment, one or more instructions for controlling the movement of object content may be stored in the memory 103 based on at least one of a preset cycle, a random cycle, and event occurrence.

The processor 101 according to an embodiment may control the overall operation of the electronic device 100 and signal flow between internal components of the electronic device 100, and may perform functions of processing data.

In an embodiment, the processor 101 may control the electronic device 100 to function by executing one or more instructions stored in the memory 103.

In embodiments, processor 101 may include single core, dual core, triple core, quad core, and multiple cores thereof.

In an embodiment, there may be one or more processors 101. For example, processor 101 may include a plurality of processors. In this case, the processor 101 may be implemented as a main processor and a sub processor.

Additionally, the processor 101 may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Video Processing Unit (VPU). Alternatively, in an embodiment, the processor 101 may be implemented in the form of a System On Chip (SoC) that integrates at least one of a CPU, GPU, and VPU. Alternatively, in an embodiment, the processor 101 may further include a Neural Processing Unit (NPU).

In an embodiment, one or more processors 101 may obtain modeling information corresponding to the projection space from modeling information about the surrounding space by executing one or more instructions.

In an embodiment, the processor 101 may obtain projection object content by correcting standard object content based on modeling information corresponding to the projection space.

In an embodiment, the processor 101 may determine the operation of projection object content based on modeling information corresponding to the projection space.

In an embodiment, the processor 101 may control the projection unit 105 to project projection content including projection object content that performs a determined operation in the projection space.

In an embodiment, one or more processors 101 may determine an operation of projection object content to be an operation corresponding to projection spatial structure information by executing one or more instructions.

In an embodiment, one or more processors 101 may determine the motion of the projection object content as the first motion in a projection space with a spatial change below a threshold by executing one or more instructions.

In an embodiment, the processor 101 may determine the motion of the projection object content to be a second motion different from the first motion in a projection space where there is a spatial change exceeding a threshold.

In an embodiment, one or more processors 101 may determine the operation of projection object content based on at least one of a preset cycle, a random cycle, and an event occurrence by executing one or more instructions.

In an embodiment, one or more processors 101 may correct at least one of color, brightness, shape, and size of standard object content based on modeling information corresponding to the projection space by executing one or more instructions.

In an embodiment, modeling information corresponding to the projection space may include projection space depth information.

In an embodiment, processor 101 may obtain user depth information.

In an embodiment, the processor 101 may correct the size of standard object content using projection space depth information and user depth information.

In an embodiment, the electronic device 100 may be rotatable about at least one of the x-axis, y-axis, and z-axis.

In an embodiment, one or more processors 101 may acquire a surrounding space image and surrounding space depth information by executing one or more instructions. In an embodiment, the processor 101 may obtain modeling information about the surrounding space based on the surrounding space image and surrounding space depth information.

In an embodiment, modeling information about the surrounding space may include color information and depth information according to the rotation angle for each axis of the electronic device 100.

In an embodiment, the processor 101 may obtain a user image and identify user feature points from the user image.

In an embodiment, the processor 101 may identify the direction in which the user's face faces based on the user's feature points. In an embodiment, the processor 101 may identify a space located in the direction where the user's face faces as the projection space.

In embodiments, the projected content may further include background content.

In an embodiment, the one or more processors 101 may determine at least one of the color and brightness of the background content based on at least one of the color and brightness of modeling information corresponding to the projection space by executing one or more instructions.

In an embodiment, the processor 101 may determine the brightness of the background content to be a minimum value.

The projection unit 105 according to the embodiment is a component that projects light for expressing an image to the outside, and may also be called a projection unit. The projection unit 105 may include various detailed components such as a light source, projection lens, and reflector.

The projection unit 105 may project images using various projection methods, for example, a cathode-ray tube (CRT) method, a liquid crystal display (LCD) method, a digital light processing (DLP) method, a laser method, etc.

The projection unit 105 may include various types of light sources. For example, the projection unit 105 may include at least one light source among a lamp, LED, and laser.

The projection unit 105 can output images in a 4:3 screen ratio, 5:4 screen ratio, or 16:9 wide screen ratio depending on the purpose of the electronic device 100 or the user's settings, and can output images in WVGA (854*480) depending on the screen ratio. ), SVGA(800*600), ), images can be output at various resolutions, such as

The projection unit 105 can perform various functions to adjust the output image under the control of the processor 101. For example, the projection unit 105 may perform functions such as zoom, keystone, and lens shift.

The projection unit 105 can automatically analyze the surrounding environment and projection environment with or without user control and perform zoom functions, keystone correction functions, focus adjustment functions, etc.

Figure 3 is an internal block diagram of the electronic device 100 according to an embodiment.

Referring to FIG. 3, in the embodiment, the electronic device 100 includes a processor 101, a memory 103, and a projection unit 105, as well as a sensing unit 106, a communication unit 107, and a user input unit 108. It may further include.

In an embodiment, the sensing unit 106 may include a sensor. The sensor may detect the state of the electronic device 100 or the state surrounding the electronic device 100, obtain raw data, and transmit it to the processor 101.

In an embodiment, the sensing unit 106 may include an image sensor 106-1. In an embodiment, the image sensor 106-1 may include a camera. The image sensor 106-1 includes a sensor such as CCD or CMOS and a lens, and can acquire an image on the screen by photographing a subject. The image sensor 106-1 can convert information about a subject illuminated by light into an electrical signal. Additionally, the image sensor 106-1 may perform one or more signal processing among AE (Auto Exposure), AWB (Auto White Balance), Color recovery, correction, Sharpening, Gamma, and Lens shading correction on the captured image. .

In an embodiment, the sensing unit 106 may include a depth sensor 106-2. The depth sensor 106-2 according to the embodiment calculates the distance between the camera and the subject using the time when the light radiated toward the subject is reflected from the subject and returns, and can obtain information about the space where the subject is located. there is. In an embodiment, the depth sensor 106-2 may recognize three-dimensional depth using one of a stereo type, a Time-Of-Flight (ToF) method, and a Structured Pattern method.

3 discloses a case where the depth sensor 106-2 is included in the electronic device 100 as a module or block separate from the image sensor 106-1, but is not limited thereto, and the depth sensor 106-2 ) may be included in the image sensor 106-1. For example, the depth sensor 106-2 may be included in a camera having a depth function, and may simultaneously acquire the distance to the subject when an image of the subject is acquired.

The sensing unit 106 may further include an acceleration sensor and/or a position sensor (eg, GPS) in addition to the image sensor 106-1 and the depth sensor 106-2. The acceleration sensor and the position sensor can obtain raw data for estimating the rotation angle of the electronic device 100. In an embodiment, the processor 101 may estimate the rotation angle of the electronic device 100 about three axes, that is, the x-axis, y-axis, and z-axis, based on raw data acquired through the sensing unit 106. there is.

The communication unit 107 according to an embodiment may include components for performing communication with other devices. For example, the communication unit 107 may include a short-range communication unit and a mobile communication unit.

In an embodiment, the short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared ray communication unit. (IrDA, infrared Data Association) communication unit, WFD (Wi-Fi Direct) communication unit, UWB (ultra wideband) communication unit, Ant+ communication unit, etc., but is not limited thereto.

In an embodiment, the BLE communication unit may transmit a BLE signal to the surroundings at all times, periodically, at random time intervals, or at preset time points.

In an embodiment, the mobile communication unit transmits and receives wireless signals to at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to voice call signals, video call signals, or text/multimedia message transmission and reception.

In an embodiment, the communication unit 107 may receive standard object content and operation information from an external server, etc. Operation information of standard object content may be information indicating various types of operations that standard object content can perform.

The user input unit 108 according to an embodiment may receive a user input for controlling the electronic device 100. The user input unit 108 includes a touch panel that detects the user's touch, a touch pad (contact capacitive type, pressure-type resistive type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, piezo effect type, etc.) , a button that receives the user's push operation, a jog wheel that receives the user's rotation operation, a jog switch, a keyboard, a key pad, and a dome switch, a microphone for voice recognition, It may include, but is not limited to, various types of user input devices including a motion detection sensor that senses motion. Additionally, when the electronic device 100 is operated by a remote controller (not shown), the user input unit 109 may receive a control signal from the remote controller.

In an embodiment, the user input unit 108 may receive a control signal from the user. A user may input a control signal to project object content using the user input unit 108. The control signal to project object content is implemented in various forms, such as when the user enters the area around the space where the electronic device 100 is located, when the user makes a specific motion, or when the user calls the object content by voice. It can be.

Additionally, the user can input a projection object content operation control signal to control the operation of the object content. In an embodiment, a user may input a projection object content operation control signal to the electronic device 100 by making a specific gesture or uttering a specific word. For example, when a user performs a specific action, such as shaking his or her hand left or right, the electronic device 100 may receive the user's motion as a control signal for controlling the projection object content operation. Alternatively, when the user utters a voice command to perform a specific action, such as “Jump!” or “Put your foot out,” the electronic device 100 may receive the voice command as a control signal that controls the operation of object content. The electronic device 100 may control the projection object content to perform a specific operation in response to a control signal according to the user's motion or voice utterance.

FIG. 4 is a diagram illustrating an operation module of the processor 101 inside the electronic device 100 according to an embodiment.

Referring to FIG. 4 , the processor 101 may include a standard object content acquisition unit 410, a modeling information acquisition unit 420, a projection space identification unit 430, and a projection content creation unit 440.

In an embodiment, the electronic device 100 may output various types of content provided by content providers. Content may include video such as still images and moving pictures, audio, subtitles, and other additional information. A content provider may be a content producer who provides various content to consumers. Content providers may include IPTV (Internet Protocol Television) service providers, OTT (Over the Top) service providers, terrestrial broadcasting stations, cable broadcasting stations, satellite broadcasting stations, etc.

In an embodiment, the electronic device 100 may receive and output various contents generated by a content provider through an external device or an external server. For example, external devices may be implemented as various types of source devices such as PCs, set-top boxes, Blu-ray disc players, mobile phones, game consoles, home theaters, audio players, USB, etc.

In an embodiment, the standard object content acquisition unit 410 may receive standard object content provided by a content provider from an external device or an external server.

Alternatively, pre-generated standard object content is stored in the memory 103 of the electronic device 100, and when the user selects one of them, the standard object content acquisition unit 410 acquires the memory 103 based on the user selection. You can obtain standard object content already stored in .

Alternatively, the user may directly create standard object content using the electronic device 100. For example, the user uses the electronic device 100 to directly draw an image of a specific type of character or avatar with a specific size or color, or acquires an image by photographing a person or object, and saves the image using the electronic device 100. ), the electronic device 100 can generate standard object content from the image.

Alternatively, the user may partially modify the standard object content already stored in the memory 103 of the electronic device 100 into a desired form and use it as standard object content.

In an embodiment, the standard object content acquisition unit 410 generates projection content using standard object content obtained from an external device, an external server, or the memory 103 of the electronic device 100, or from an image input by the user. It can be transmitted to unit 440.

In an embodiment, the modeling information acquisition unit 420 may acquire modeling information about the surrounding space.

In an embodiment, the electronic device 100 may acquire an image of the surrounding space through the image sensor 106-1. In an embodiment, since the electronic device 100 can rotate more than a predetermined angle in each of the x, y, and z axes, the image sensor 106-1 mounted on the electronic device 100 also rotates together with the electronic device 100. While doing this, you can acquire images from multiple angles of the surrounding space.

In an embodiment, the electronic device 100 may acquire surrounding space depth information through the depth sensor 106-2. In an embodiment, surrounding space depth information may mean distance information to an environment or structure in the surrounding space.

In an embodiment, the modeling information acquisition unit 420 models the surrounding space based on the surrounding space image acquired through the image sensor 106-1 and the surrounding space depth information acquired through the depth sensor 106-2. Information can be obtained.

In an embodiment, the modeling information acquisition unit 420 may shape the shape of the surrounding space by connecting the environment of the surrounding space, the surface of an object placed in the surrounding space, the position of an object, geometric information, etc., with points, lines, or surfaces. You can. In an embodiment, the modeling information acquisition unit 420 performs rendering on numerous lines or points generated for the environment or objects in the surrounding space to create colors, volume, and texture similar to the actual environment or actual objects. You can have it.

In an embodiment, modeling information about the surrounding space may include color information and depth information according to the rotation angle for each axis of the electronic device 100.

In an embodiment, the modeling information acquisition unit 420 may transmit modeling information about the surrounding space to the projection content creation unit 440.

In an embodiment, the projection space identification unit 430 may identify the projection space. In an embodiment, the projection space identification unit 430 may detect user feature points from the user image. A user feature point may be information for identifying a part of the user's body, such as the user's head, ears, neck, shoulders, or face.

In an embodiment, the projection space identification unit 430 may generate a figure modeling the user's body by connecting user feature points detected from the image. For example, the projection space identification unit 430 may generate a hexahedral shape that includes the back of the user's head at the center and both ears of the user at the ends of the shape. In an embodiment, the projection space identification unit 430 may identify the direction in which the user's face is facing based on a figure modeling the user's body. Additionally, in an embodiment, the projection space identification unit 430 identifies which axis and how much the figure modeling the user's body is tilted, and accordingly identifies which axis and how much the direction the user's face faces is tilted. can do.

In an embodiment, the projection space identification unit 430 may identify a space located in the direction where the user's face faces as the projection space. In an embodiment, the projection space may be displayed as a rotation angle range for each axis of the electronic device 100. Alternatively, in embodiments, the projection space may be expressed as a range of unique coordinate values of the identified projection space.

In an embodiment, the projection space identification unit 430 may transmit information indicating the projection space to the projection content creation unit 440.

In an embodiment, the projection content creation unit 440 may generate projection content to be projected through the projection unit 105. In an embodiment, the projection content creation unit 440 may identify modeling information corresponding to the projection space received from the projection space identification unit 430 among the modeling information about the surrounding environment received from the modeling information acquisition unit 420. You can.

In an embodiment, the projection content generator 440 may include an object content shape correction unit 441, an object content operation determination unit 443, and a background content generator 445.

In an embodiment, the object content shape correction unit 441 may correct the shape of the standard object content received from the standard object content acquisition unit 410 based on modeling information corresponding to the projection space. In an embodiment, the object content shape correction unit 441 may correct the shape, such as color, brightness, shape, and size, of standard object content based on modeling information corresponding to the projection space.

In an embodiment, the object content shape correction unit 441 may independently correct color, brightness, shape, size, etc. for each of the plurality of sub-areas constituting standard object content. In an embodiment, the object content shape correction unit 441 may divide standard object content into a plurality of sub-areas based on modeling information corresponding to the projection space. For example, the object content shape correction unit 441 may divide the standard object content into a plurality of sub-areas for each color-differentiated area in the projection space, and correct the divided sub-areas differently according to the color of the projection space.

If there is an area in the projection space that is darker than the surrounding area, the area of the object content projected to the dark projection space appears relatively darker than other areas of the projected object content. Additionally, if there is an area in the projection space that is brighter than the surrounding surroundings, the area of the object content projected in the bright projection space appears relatively brighter than other areas of the projected object content. In other words, when a dark area and a bright area exist together in the projection space, even if object content of the same brightness is projected, the object content appears partly bright and partly dark depending on the brightness of the projection space.

Similarly, even if the object content is the same, the color or brightness of the object content may appear different depending on the color of the projection space due to phenomena such as color contrast.

Additionally, even if the object content is the same, the size or shape of the object may appear differently distorted depending on the depth information of the projection space. For example, when the projection space includes a protruding plane, the object content area projected onto the protruding plane appears larger than the object content area projected onto the non-protruding area.

Additionally, if the projection space is not a smooth plane but is uneven, the object content projected onto the uneven space appears distorted.

Accordingly, in the embodiment, the object content shape correction unit 441 distorts the parts that make up one object content into different colors, brightness, sizes, shapes, etc., according to different brightness, color, depth information, etc. of the projection space. To prevent this from appearing, each area of the standard object content can be corrected differently based on modeling information about the projection space.

In an embodiment, the object content shape correction unit 441 may differently adjust color, brightness, shape, size, etc. for each pixel of standard object content.

In an embodiment, the object content shape correction unit 441 corrects the color, brightness, shape, size, etc. differently for each of the plurality of sub-areas, so that the overall color, brightness, shape, size, etc. of the corrected and projected object content are unified. It can be displayed in color, brightness, size, shape, etc.

In an embodiment, the object content operation determination unit 443 may determine the operation of projected object content. In an embodiment, the object content operation determination unit 443 may determine the operation of the projection object content based on modeling information corresponding to the projection space. In an embodiment, modeling information corresponding to the projection space may include projection space structure information.

In an embodiment, the object content operation determination unit 443 may determine the operation of the projection object content as an operation corresponding to the projection space structure information.

In an embodiment, the projected spatial structure information may include spatial change information.

In an embodiment, the object content motion determination unit 443 determines the motion of the projection object content as the first motion in a projection space with a spatial change below the threshold, and determines the projection object content as the first motion in a projection space with a spatial change exceeding the threshold. The content operation may be determined as a second operation different from the first operation.

For example, based on the depth value difference in the projection space, the object content operation determination unit 443 allows the projection object content to perform different operations in an area with a depth value difference greater than a threshold value and an area with a depth value difference less than the threshold value. You can control it.

In an embodiment, when creating standard object content, the content provider determines the color, design, size, type, etc. of the standard object content, while also determining the types of operations that the standard object content can perform, that is, motion information. You can decide. For example, if the standard object content is a ladybug, the content provider may determine the crawling motion of the ladybug as an motion that the standard object content can perform. Additionally, the content provider may determine in advance the operations that standard object content can perform in an area where the difference in depth values of the projection space is greater than the threshold. For example, in the above example, the content provider may decide to perform a flying motion by spreading its wings when the ladybug moves through an area where the difference in depth values is greater than a threshold.

In an embodiment, the object content operation determination unit 443 may determine the operation of the object content using operation information about predetermined operations that standard object content can perform.

In an embodiment, the background content generator 445 may generate background content. In an embodiment, the background content may mean the remaining area of the projection content excluding the projection object content. In an embodiment, the background content generator 445 may identify modeling information corresponding to the projection space received from the projection space identification unit 430 among the modeling information about the surrounding environment received from the modeling information acquisition unit 420. You can.

In an embodiment, the background content generator 445 may generate background content based on modeling information corresponding to the projection space. In an embodiment, the background content generator 445 may determine at least one of the color and brightness of the background content based on at least one of the color and brightness of modeling information corresponding to the projection space. For example, the background content generator 445 may generate background content with the same color and brightness as the color and brightness of the projection space. In this case, the background content is projected with colors and brightness similar to the actual space, so the user cannot clearly distinguish the projected background content from the actual space.

In an embodiment, the background content generator 445 may determine the minimum brightness value of the background content. If the background content is projected at minimum brightness, it cannot be clearly identified, so it can have the same effect as if only the projected object content is projected.

In an embodiment, the projection content generator 440 generates object content with a corrected shape generated by the object content shape correction unit 441, that is, projection object content, and the background content generated by the background content generator 445. Projection content including content can be created and projected into the projection space.

In an embodiment, projection object content included in projection content may perform an operation determined by the object content operation determination unit 443.

FIG. 5 is a diagram for explaining rotation of the electronic device 100 according to an embodiment.

Referring to FIG. 5 , the electronic device 100 may be mounted on a fixed surface 500 such as a wall, ceiling, or stand. In an embodiment, the electronic device 100 may be mounted on the fixing surface 500 through the support part 510. The support unit 510 may be a component used to fix the electronic device 100 in a specific position. In an embodiment, the support portion 510 may be connected to a bracket 520. The bracket 520 may also be called a mount. The bracket may be a component that connects the support portion 510 and the electronic device 100. In an embodiment, the electronic device 100 may be connected to the bracket 520 in a rotatable manner.

In an embodiment, the projection unit 530 may be disposed at one end of the electronic device 100.

Referring to FIG. 5, assuming that there is a projection surface perpendicular to the projection direction in which the projection unit 530 projects projection content, the projection direction of light projected perpendicular to the projection surface or the direction opposite to the projection of light is the z-axis. It can be defined as: Additionally, a direction parallel to the projection surface and perpendicular to the z-axis can be defined as the y-axis, and a direction parallel to the projection surface and perpendicular to the z-axis and y-axis can be defined as the x-axis.

In an embodiment, the electronic device 100 may rotate along each axis. For example, the electronic device 100 can rotate up to 180 degrees along the x-axis and y-axis parallel to the fixing surface 500. Additionally, in an embodiment, the electronic device 100 may rotate up to 360 degrees along the z-axis.

FIG. 6 is a diagram to explain how the electronic device 100 acquires modeling information about its surroundings, according to an embodiment.

Referring to FIG. 6, the electronic device 100 can be rotated by being placed in a rotatable form on a wall or ceiling. However, this is just an example, and the electronic device 100 may be placed on the floor or may be mobile rather than fixed in a specific location.

In an embodiment, when the electronic device 100 rotates along each axis, the image sensor 106-1 and the depth sensor 106-2 included in the electronic device 100 also rotate along the axis.

In an embodiment, the electronic device 100 may obtain an image of the surrounding environment or objects using the image sensor 106-1 provided in the electronic device 100.

In an embodiment, the electronic device 100 may acquire depth information about the surrounding environment or an object using the depth sensor 106-2 provided in the electronic device 100.

In an embodiment, the electronic device 100 may model the surrounding space using an image of the surroundings and depth information about the surroundings.

As shown in FIG. 6, the electronic device 100 may obtain modeling information 615 and 625 about the surrounding space by modeling the actual surrounding space 610 and 620.

In embodiments, the electronic device 100 may model the surrounding space in various ways. In an embodiment, the electronic device 100 may express the shape of the surrounding space as basic geometric elements such as points, straight lines, circles, and arcs using the Wireframe Modeling method, which is a method of expressing three-dimensional objects with lines. Wireframe modeling is a method of expressing the shape using the characteristic lines (wireframe) and end points of an object, and the shape of the model can be modified by modifying the lines and points. In an embodiment, the electronic device 100 may generate a realistic image by modeling the surrounding space using a wireframe modeling method and performing a rendering operation to color the surface created when lines meet.

Alternatively, in an embodiment, the electronic device 100 may express a three-dimensional object as a surface using a surface modeling method. The Surface Modeling method is a technique that allows the expression of complex curved surfaces by adding surface information to the Wireframe Modeling technique. This modeling method is a technique that decomposes a three-dimensional object into two-dimensional surface components and can create a three-dimensional shape by combining multiple surfaces. In this method, only surface data is obtained without information about the interior of the object. Since information about the inside of the object is not taken into account, computational speed and memory capacity can be used less.

In an embodiment, the electronic device 100 acquires the coordinates of points forming the surrounding space, a list of curve equations representing a line, a list of curved equations, information on the relationship between points, lines, and curved surfaces, and uses a Surface Modeling method based on this. You can model the surrounding space using .

In an embodiment, the electronic device 100 may model space using the Polygon Modeling method, which is one of the Surface Modeling methods. Polygon Modeling is a method that uses points (vertex) as the basic unit to create a three-dimensional form by combining points (edges) and lines (faces) connecting lines. In the case of the polygon modeling method, several polygon-shaped polygons are gathered together to form one character or object. However, it is not limited to this, and the electronic device 100 may acquire modeling information about the surrounding space using the Nurbs (Non-Uniform Rational B-Spline Modeling) modeling method among the Surface Modeling methods or the Subdivision Surface Modeling method. It may be possible.

FIG. 7 is a diagram for explaining how the electronic device 100 identifies a projection direction, according to an embodiment.

In an embodiment, the electronic device 100 may photograph the user using the image sensor 106-1. For example, when the user is not looking at the electronic device 100 but is looking in the opposite direction, the image sensor 106-1 may obtain a user image by photographing the user from the back of the user. The user image acquired at this time may include the back of the user's head, the back of both ears of the user, the back of the user's neck, and shoulders.

In an embodiment, the electronic device 100 may analyze a user image obtained by photographing a user. In an embodiment, the electronic device 100 may detect user feature points from a user image. The user feature point may be a key point for each body part to identify the body part.

In an embodiment, the electronic device 100 may connect identified user feature points and generate a figure connecting the user feature points. The figure connecting user feature points may be a two-dimensional flat figure or a three-dimensional solid figure.

FIG. 7 shows that the electronic device 100 generates a hexahedron shape by connecting user feature points detected from the user image. However, this is one embodiment, and the electronic device 100 may generate various types of figures based on user feature points.

In an embodiment, the electronic device 100 may identify the degree of inclination of a figure created by connecting user feature points. If the user image is acquired while the user is looking straight ahead, the three-dimensional figure created by connecting the user feature points may be in the form of an upright reference figure without tilt.

If the user does not look straight ahead but slightly tilts his head to either the left or right or raises his face to look at the ceiling, the three-dimensional figure created by connecting the user's feature points is tilted in one direction, unlike the reference figure. It can be.

In an embodiment, the electronic device 100 may compare a three-dimensional figure created by connecting user feature points with a reference figure to identify the direction in which the front of the three-dimensional figure faces and the direction in which the three-dimensional figure is tilted. In an embodiment, the electronic device 100 may identify the direction of the user's face based on the tilt direction of the three-dimensional figure.

In an embodiment, the electronic device 100 may identify the direction in which the user's face faces as the projection direction, and identify the space located in the direction in which the user's face faces as the projection space.

In an embodiment, the electronic device 100 may project projection content to a projection space located in the direction where the user's face faces.

In an embodiment, the electronic device 100 may re-identify the projection direction when the user's viewing direction changes by more than a threshold. For example, when a user looks at a first wall in a space and then turns his face to look at a second wall facing the first wall, the electronic device 100 acquires a new user image and matches the user feature points obtained therefrom. Based on this, the direction the user's face is facing can be re-identified.

In an embodiment, the electronic device 100 may identify a space located on the second wall toward which the user's face faces as a new projection space. When the projection space changes, the electronic device 100 may control projection object content to operate in a predefined order. For example, when the projected object content is a whale, the electronic device 100 can control the whale to perform an action of spouting water in a new projection space.

FIG. 8 is a diagram for explaining how the electronic device 100 corrects the shape of object content based on modeling information corresponding to the projection space, according to an embodiment.

In an embodiment, the electronic device 100 may obtain modeling information about the surrounding space. In an embodiment, the electronic device 100 may identify the direction in which the user's face faces and identify the space located in the direction in which the user's face faces as the projection space.

In an embodiment, the electronic device 100 may identify modeling information corresponding to the projection space, which is the space toward which the user's face is facing, from among modeling information about the surrounding space.

Reference numeral 810 in FIG. 8 represents a diagram in which projection object content is projected in a projection space. Figure 8 shows, as an example, a case where the projected object content is a cat. In an embodiment, the electronic device 100 may acquire standard object content having the shape of a cat. In an embodiment, standard object content may be expressed as a three-dimensional shape with width, height, and length, as shown at reference numeral 820. Additionally, standard object content can be expressed as having color information and luminance information for each pixel.

In an embodiment, modeling information corresponding to the projection space may include color information and luminance information according to the rotation angle for each axis of the electronic device 100. For example, if two projection points included in an area indicated by A in the projection space of reference numeral 810 are respectively referred to as first point 831 and second point 832, as shown in reference numeral 830, the first point 831 and the second point 832 may be expressed as color information (r, g, b) and luminance information (Y), respectively. The first point 831 has (r, g, b, Y) at (200, 180, 195, 200), and the second point 832 has (r, g, b, Y) at (50, 40, 32, 48).

Among the projection points, the first point 831 has a color close to white and is bright, while the second point 832, unlike the first point 831, has a deep and dark color and brightness.

In an embodiment, the electronic device 100 may identify a corresponding point of standard object content projected to a projection point in the projection space. For example, the electronic device 100 identifies the first point 841 and the second point 842, which are corresponding points of the standard object content projected to the first point 831 and the second point 832, respectively. can do.

In an embodiment, standard object content is content created based on when the color, brightness, depth information, etc. of the projection space have standard values, so when the color, brightness, spatial structure, etc. of the projection space do not have standard values, The size, shape, color, and brightness of the standard object content projected into such a projection space also have a different form from the standard object content.

In an embodiment, when the color and luminance of the projection space where the standard object content is projected are standard color and standard luminance, the electronic device 100 may project the standard object content as is without correcting the color or luminance of the standard object content. You can. However, when the color or luminance of the projection space is not the standard color and standard luminance, the electronic device 100 corrects the color or luminance of the standard object content according to the color or luminance of the projection space, resulting in the projection projected into the projection space. Object content can have the color or luminance of standard object content.

If the projection space where standard object content is projected includes areas with different colors or luminance, and if the standard object content is projected as is without correcting the color or luminance of the standard object content, the different colors or luminance of the projection space Even though the standard object content projected over the area is one standard object, the areas of the projected object content also appear to have different colors or luminance due to the different colors or luminance of the projection area.

For example, in the example above, the color of the first point 831 among the projection points is white and bright, while the color of the second point 832 is dark and bright, so the first point 831 and the second point 832 are Even if the color and luminance of the first point 841 and the second point 842 of the projection object content projected to 832 are the same, when projected into the projection space, the first point 841 of the standard object content is the first point 841 of the standard object content. It is expressed whiter and brighter than point 2 842, and the second point 842 is expressed darker and darker.

In an embodiment, the electronic device 100 may correct the color, brightness, shape, size, etc. of projection object content based on modeling information corresponding to the projection space. In an embodiment, the electronic device 100 uses the color information and luminance information of the projection point in the projection space to correct the color information and luminance information of the corresponding point of the standard object content projected to the projection point to produce projection object content. can be created.

In an embodiment, the electronic device 100 displays the color information and luminance information of the projection point even when the color information and luminance information of the first point 841 and the second point 842 of the standard object content have the same value. Accordingly, the color information and luminance information of the corresponding point of the standard object content can be corrected to different values.

For example, assuming that the first point 841 and the second point 842 of the standard object content have the same color information and luminance information, in the embodiment, the electronic device 100 is connected to the first point 842, which is the projection point. Using the color information and luminance information of (831), the color information and luminance information of the first point (841) of the standard object content projected to the first point (831) are converted to (80, 60, 10, 40). It can be corrected. In addition, the electronic device 100 uses the color information and luminance information of the second point 832, which is the projection point, to display the color information of the second point 842 and the color information of the second point 842 of the standard object content projected to the second point 832. The luminance information can be corrected as (120, 80, 15, 65).

That is, in the embodiment, the electronic device 100 changes the color and luminance of the first point 841 of the standard object content projected to the first point 831, which is white in color and bright in brightness, to a darker and darker color among the projection points. By correcting the color and luminance of the second point 842 of the projection object content projected at the second point 832, where the color is darker and darker, to whiter and brighter, the first point of the projection object content projected in the projection space 841 and the second point 842 can be expressed with unified color and luminance.

Likewise, a projection point included in an area indicated by B in the projection space of reference numeral 810 may be expressed as color information (r, g, b) and luminance information (Y), respectively. In an embodiment, it can be seen that the upper and lower areas of the area marked B have different luminance information. In an embodiment, when the electronic device 100 projects standard object content across the upper and lower areas included in area B, the luminance information at each point of the standard object content may be differently corrected depending on the projection point.

In an embodiment, the electronic device 100 corrects the luminance of a point of projection object content projected to a relatively brighter point among projection points to have a darker luminance, and corrects the luminance of a point of projection object content projected to a relatively brighter point among projection points to have a darker luminance. By correcting the luminance of a point to have a brighter luminance, different points of the projection object content projected in the projection space can be expressed as if they have the same luminance.

FIG. 9 is a diagram for explaining how the electronic device 100 corrects the shape of object content based on modeling information corresponding to the projection space, according to an embodiment.

In an embodiment, the electronic device 100 may identify modeling information corresponding to the projection space among modeling information about the surrounding space.

In an embodiment, modeling information corresponding to the projection space may include depth information. In an embodiment, the electronic device 100 may adjust the size of standard object content using depth information of the projection space.

FIG. 9 shows two areas, area A and area B, respectively, included in the projection space indicated by reference numeral 810 in FIG. 8 .

In an embodiment, each point included in area A and area B may have different depth values. For example, area A may include an upper wall area and a lower floor area. In an embodiment, the wall area and the floor area included in area A may have different depth values. That is, because the distance from the electronic device 100 to the wall is longer than the distance from the electronic device 100 to the floor, the wall area has a larger depth value. Additionally, the floor area has a larger depth value as it moves closer to the boundary point with the wall area, and has a smaller depth value as it moves away from the boundary point.

Even though the projection space where standard object content is projected includes areas with different depth values, if the standard object content is projected as is without correcting its size, the standard object content is projected across areas with different depth values in the projection space. Even though object content is a single standard object, it may appear distorted and have different sizes for each area due to different depth values of the projection area.

For example, in the above example, when standard object content is projected on the wall and floor as is even though the depth value of the wall is larger and the depth value of the floor is smaller, the standard object content projected on the wall is due to the distance from the wall. The projection point of is expressed as having a smaller value.

In an embodiment, the electronic device 100 may correct the size of projection object content based on modeling information corresponding to the projection space. In an embodiment, the electronic device 100 may use the depth value of the projection point in the projection space to correct the size of the corresponding point of the standard object content projected to the projection point.

In an embodiment, the electronic device 100 may adjust the size of a corresponding point of a standard object projected to a point having a large depth value to be larger, and adjust the size of a corresponding point having a small depth value to be smaller. That is, in the above example, the electronic device 100 may correct the size of the standard object content projected onto the wall area to be larger. In addition, the electronic device 100 corrects the size of the standard object content projected to the floor differently depending on the depth value, so that the size of the standard object content projected to that point becomes larger as it moves from the floor to the boundary point with the wall. The size of the standard object content projected to that point can be corrected to be smaller as the point moves away from the boundary point with the wall.

Likewise, the electronic device 100 may differently correct the sizes of corresponding points of standard objects projected to points with different depth values in area B.

Accordingly, the overall shape and size of the projection object content projected in the projection space can be expressed as having a value that is the same as the overall shape and size of the standard object content or is proportional to the size of the standard object content.

In an embodiment, the electronic device 100 may acquire user depth information. User depth information may be information indicating the distance between the electronic device 100 and the user. In an embodiment, the electronic device 100 may obtain the distance to the user using the depth sensor 106-2.

In an embodiment, the electronic device 100 may correct the size of standard object content using the user's depth information and projection space depth information. In an embodiment, when the distance between the user and the projection space is short, the electronic device 100 may correct the overall size of the standard object content so that it increases according to the distance. In an embodiment, when the distance between the user and the projection space is long, the electronic device 100 may correct the size of the standard object content so that the overall size of the standard object content appears smaller.

As such, according to the embodiment, the electronic device 100 uses modeling information about the projection space, user depth information, etc. to correct and project the color, brightness, size, etc. of standard object content, thereby providing a more realistic projection to the user. Object content can be provided.

FIG. 10 is a diagram illustrating how the electronic device 100 corrects the shape of standard object content based on modeling information corresponding to the projection space, according to an embodiment.

The projection space may be a flat plane, but may also have a three-dimensional effect. Distortion that occurs because the projection space is not flat can be corrected through geometry correction. Geometry correction is a method that reverses the principle of image projection.

Reference numeral 1010 in FIG. 10 indicates a case where the electronic device 100 projects standard object content into the projection space as is without geometry correction. If the projection space is not flat and has a depth value, the standard object content projected into the projection space is projected in a distorted form due to the depth value of the projection space. This distortion occurs because when pixels included in standard object content are projected, they are not projected exactly where they should be, but are distorted due to the depth value of the projection space.

In an embodiment, the electronic device 100 may perform geometry correction by obtaining information that the standard object content is distorted and using this inversely to correct the standard object content so that it is distorted in the opposite direction. In an embodiment, the electronic device 100 may sample some of the pixels included in standard object content and modify the location where the sampled pixels are projected. In an embodiment, the electronic device 100 may perform geometry correction by interpolating and compensating between pixels with modified positions.

Reference numeral 1020 in FIG. 10 indicates object content that has undergone geometry correction in order for the electronic device 100 to project standard object content without distortion in a projection space with different depth values.

Reference numeral 1030 in FIG. 10 indicates a case where the electronic device 100 projects object content on which geometry correction has been performed into a projection space. Unlike reference numeral 1010, object content on which geometry correction has been performed can be projected as a standard object content shape without distortion in a projection space with a depth value.

FIG. 11 is a diagram explaining how the electronic device 100 determines the movement of object content according to a projection space, according to an embodiment.

In an embodiment, the electronic device 100 may obtain modeling information corresponding to the projection space. In an embodiment, modeling information corresponding to the projection space may include projection space structure information.

In an embodiment, the electronic device 100 may determine the operation of projection object content as an operation corresponding to projection space structure information. In an embodiment, the projected spatial structure information may include spatial change information.

In an embodiment, the electronic device 100 may determine the operation of projection object content differently in a projection space with a spatial change below the threshold and a projection space with a spatial change exceeding the threshold.

In an embodiment, a projection area where there is a spatial change greater than a threshold may mean a place where there is a spatial change of a certain size or more along at least one of the x-axis, y-axis, and z-axis.

In FIG. 11 , reference numeral 1110 indicates a case where projection object content moves from position 1 to position 2. As shown at reference numeral 1120, positions 1 and 2 may be positions where there is a change greater than a threshold value in the x-axis, y-axis, and z-axis directions, respectively.

In an embodiment, the electronic device 100 may obtain coordinate values for each of location 1 and location 2, and identify whether there is a change greater than a threshold value between the two locations based on the difference in coordinate values.

In an embodiment, motion information may be divided into a first motion and a second motion according to spatial changes in the projection space.

In an embodiment, the first operation may refer to a general operation that object content can perform when the change in projection space is less than or equal to a threshold. For example, if the standard object content is a dog, the first motion may include a motion of standing in one place and wagging its tail, a motion of crouching down, a motion of walking, etc.

In an embodiment, the second operation may refer to an operation performed when the projection object content moves in a projection area where the spatial change exceeds the threshold. In an embodiment, the second operation may be a different operation than the first operation.

In an embodiment, the second motion may be divided into different motions depending on which direction the spatial change occurs. For example, the second motion may be divided into different motions depending on whether there is a spatial change in the x-axis, y-axis, or z-axis.

In an embodiment, if the y-axis represents height, the space between the floor and the top of the stairs may be a space that changes in the y-axis direction. In Figure 11, it is assumed that the height difference between the floor and the top of the stairs is greater than a critical value. In an embodiment, the electronic device 100 may identify that there is a spatial change in the floor and stairs on the y-axis that is greater than a threshold value, and may control the projection object content to perform the second operation accordingly. The second action may be, for example, an action of jumping up a space with a height difference.

In an embodiment, if the x-axis represents width, that is, the horizontal axis, the space between two chairs that are more than a threshold distance apart in the horizontal direction may be a space where there is a change in the x-axis direction. In an embodiment, the electronic device 100 may control the projection object content to perform a second operation when the projection object content moves between points that are separated by a certain distance or more in the x-axis direction. The second action may be, for example, a jumping action.

In an embodiment, if the z-axis represents a depth difference, spaces spaced apart in the z-axis direction by more than a threshold may be spaces where there is a change in the z-axis direction. In an embodiment, the electronic device 100 may control the projection object content to perform a second operation when the projection object content moves between spaces having a depth value difference greater than or equal to a threshold value. The second motion may be, for example, a jumping motion while the projected object content moves in a spatial direction with a small depth value.

In an embodiment, the second operation may be further divided according to the degree of spatial change. For example, the second operation may be divided into different operations depending on when the spatial change is above the threshold and below the maximum value and when the spatial change is above the maximum value.

In an embodiment, when the degree of spatial change is very large, such as when the height of the stairs in the y-axis direction is very high and the height difference between the floor and the top of the stairs is greater than a preset maximum value, the electronic device 100 may display the projection object content. Among the second actions, instead of the jumping action, the user may be controlled to perform an action of jumping up the stairs and then falling back to the original position, or the projection object content may be controlled to move around the bottom of the stairs.

In an embodiment, when the degree of spatial change is very large, such as when the distance between two chairs in the You can control it to jump down to the floor, and then jump back up onto the chair next to it.

In an embodiment, when the degree of spatial change is very large, such as when the depth difference in the z-axis is greater than the maximum value, the electronic device 100 may cause the projection object content to perform a long jump or jump instead of a jumping motion among the second motions. I have control to perform the action.

Referring to FIG. 11, the electronic device 100 determines that the space between position 1 and position 2 is a space where there is a change greater than a threshold value, and the projection object content is displayed in an event such as running up stairs as shown in reference numeral 1130. You can control movement between spaces with 2 movements.

As such, according to the embodiment, the electronic device 100 controls the movement method of the projection object content in various ways in consideration of the amount of change in the projection space, so that the projection object content acts as if it actually recognizes and operates in the real space. You can.

FIG. 12 is a diagram explaining how the electronic device 100 recognizes the amount of change in the projection space and controls the operation of projection object content accordingly, according to an embodiment.

In an embodiment, the electronic device 100 may control the electronic device 100 to operate in a second operation when a structure is located in a projection space where projection object content moves. However, the electronic device 100 is not limited to this, and the electronic device 100 may control the motion or movement direction of the projection object content so that the projection object content operates in the first motion but does not collide with the structure.

Referring to FIG. 12 , in an embodiment, the electronic device 100 may obtain modeling information corresponding to the projection space from modeling information about the surrounding environment. The drawing shown above in FIG. 12 shows a case where a structure such as a television or a chest of drawers is located in the projection space. In an embodiment, the electronic device 100 may identify the location of the structure from modeling information corresponding to the projection space.

As shown in FIG. 12, when the projected object content is a whale, in an embodiment, the electronic device 100 controls the whale to move around the structure to prevent the whale from colliding with the structure as it moves along the wall. You can control the direction of movement.

Or, in an embodiment, as shown in the figure below in FIG. 12, when the whale moves between walls, that is, around a corner, the electronic device 100 adjusts the whale's movement direction so that the whale moves as if it were going around the corner. You can control it.

As such, according to the embodiment, the electronic device 100 controls the direction of movement of the projection object content in consideration of the amount of change in the projection space, thereby preventing the projection object content from colliding with a structure located in the projection space. Accordingly, the user can experience a more satisfactory augmented reality by recognizing the projected object content as if it recognizes and operates in real space.

FIG. 13 is a diagram explaining how the electronic device 100 adjusts the size of background content, according to an embodiment.

In an embodiment, when the electronic device 100 acquires standard object content, it may correct the standard object content to generate projection object content and also generate background content around the projection object content.

FIG. 13 illustrates that when the electronic device 100 outputs the same projection object content, background content included in the projection content along with the projection object is adjusted differently. It can be seen that the background content on the left side of FIG. 13 is relatively large, while the background content on the right side of FIG. 13 is small.

In an embodiment, the electronic device 100 may control projection object content to operate within the boundary of the background content. Therefore, the larger the background content, the larger the projected object content can move within a wider area. On the other hand, when the background content is small, the movement radius of the background content is narrowed, but the background content has the advantage of being less visible to the user.

In an embodiment, the electronic device 100 may determine the size of background content.

In an embodiment, the electronic device 100 may identify the structure of the surrounding space based on modeling information about the surrounding space and determine the size of the background content to be larger or smaller accordingly. For example, when there are not many structures in the surrounding space and many simple, wide planes, the electronic device 100 can project background content in a large size. On the other hand, when the surrounding space has many structures and is complex, the electronic device 100 may project the background content small so that the background content is not easily visible to the user.

In an embodiment, the electronic device 100 may determine the size of the background content according to the specifications or model of the electronic device 100. For example, if the specifications of the electronic device 100 are specifications that allow the electronic device 100 to freely rotate various angles, for example, 360 degrees, for each axis, the electronic device 100 may set the size of the background content to be large. Alternatively, you can set the size of the background content to be small. The electronic device 100 can output projection content by variously adjusting the screen size, aspect ratio, resolution, etc.

In an embodiment, when the specifications of the electronic device 100 do not have a rotation function or have restrictions on the rotation angle or rotation movement, the electronic device 100 sets the size of the background content to be large so that the projection object can be displayed within the wide background content. You can control it to move.

As such, according to the embodiment, the electronic device 100 may adjust the size of the background content to different sizes according to various standards and project it.

FIG. 14 is a flowchart illustrating a method of obtaining modeling information about surrounding space, according to an embodiment.

Referring to FIG. 14, the electronic device 100 may acquire an image of the surrounding space (step 1410). In an embodiment, the electronic device 100 may acquire an image of the surroundings of the electronic device 100 using the image sensor 106-1.

In an embodiment, the electronic device 100 may acquire surrounding space depth information (step 1420). In an embodiment, the electronic device 100 may obtain depth information about the surroundings of the electronic device 100 using the depth sensor 106-2.

In an embodiment, the electronic device 100 may obtain modeling information about the surrounding space (step 1430). In an embodiment, the electronic device 100 may acquire modeling information about the surrounding space by virtually shaping the surrounding space based on an image of the surroundings and depth information about the surroundings. In an embodiment, modeling information about the surrounding space may include color information, luminance information, and depth information according to the rotation angle for each axis of the electronic device 100.

FIG. 15 is a flowchart illustrating a method of operating the electronic device 100 according to an embodiment.

In an embodiment, the electronic device 100 may identify a projection space. In an embodiment, the projection space may be in the direction the user's face faces.

In an embodiment, the electronic device 100 may acquire a user image using the image sensor 106-1 and identify user feature points from the user image. In an embodiment, the electronic device 100 may identify the direction in which the user's face faces based on the user's characteristic points. In an embodiment, the electronic device 100 may identify a space located in the direction where the user's face faces as the projection space.

In an embodiment, the electronic device 100 may obtain modeling information corresponding to the projection space from modeling information about the surrounding space (step 1510). The electronic device 100 may obtain modeling information on a space corresponding to the projected space range among the surrounding spaces.

In an embodiment, the electronic device 100 may acquire standard object content.

In an embodiment, the electronic device 100 may obtain projection object content by correcting the shape of standard object content based on modeling information corresponding to the projection space (step 1520).

In an embodiment, the electronic device 100 may correct at least one of color, brightness, shape, and size of standard object content based on modeling information corresponding to the projection space.

In an embodiment, the electronic device 100 may determine the operation of projection object content based on modeling information corresponding to the projection space (step 1530). In an embodiment, the electronic device 100 may determine the operation of the projection object content in various ways depending on the projection space structure and the degree of change in the structure.

In an embodiment, the electronic device 100 may generate background content and generate projection content including projection object content and background content.

In an embodiment, the electronic device 100 may project projection content in which the projection object content performs the determined operation into the projection space (step 1540).

FIG. 16 is a flowchart illustrating different control of the operation of projection object content based on a difference in depth value of the projection space, according to an embodiment.

In an embodiment, the electronic device 100 may control the operation of the projection object content according to the structure of the space in which the projection object content moves.

In an embodiment, the electronic device 100 may control the operation of projection object content differently depending on whether the change in projection space is greater than a threshold.

Referring to FIG. 16, in an embodiment, the electronic device 100 may determine whether object content moves in space with a depth value difference smaller than a threshold (step 1610).

In an embodiment, when the electronic device 100 determines that the object content moves in space with a depth value difference smaller than the threshold, the electronic device 100 may determine the motion of the object content as the first motion (step 1620).

In an embodiment, if the electronic device 100 determines that the object content is not moving in space with a depth value difference smaller than the threshold, the electronic device 100 may determine the second motion of the object content (step 1630).

In an embodiment, the electronic device 100 may control the operation of the projection object content using the determined operation (step 1640).

Methods and devices for operating electronic devices according to some embodiments may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism, and includes any information delivery medium.

In addition, the electronic device and its operating method according to the above-described embodiment of the present disclosure include obtaining modeling information corresponding to the projection space from modeling information about the surrounding space, based on the modeling information corresponding to the projection space, Obtaining projection object content by correcting the shape of the object content, determining an operation of the projection object content based on modeling information corresponding to the projection space, and performing the determined operation in the projection space. A program for implementing a method of operating an electronic device, including the step of projecting projection content including projection object content, may be implemented as a computer program product including a computer-readable recording/storage medium.

A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. A computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store or between two user devices (e.g. smartphones). It may be distributed in person or online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

Claims (20)

In the electronic device 100,
projection unit 105;
a memory 103 that stores one or more instructions; and
Comprising one or more processors (101) executing one or more instructions stored in the memory,
The one or more processors execute the one or more instructions,
Obtain modeling information corresponding to the projection space from modeling information about the surrounding space,
Based on the modeling information corresponding to the projection space, correct the shape of the standard object content to obtain projection object content,
Based on modeling information corresponding to the projection space, determine an operation of the projection object content,
An electronic device that projects, through the projection unit, projection content including the projection object content that performs the determined operation into the projection space. The method of claim 1, wherein the modeling information corresponding to the projection space includes projection space structure information,
The electronic device wherein the one or more processors determine an operation of the projection object content to be an operation corresponding to the projection space structure information by executing the one or more instructions. The method of claim 2, wherein the projection spatial structure information includes spatial change information,
The one or more processors execute the one or more instructions,
In a projection space where there is a spatial change below a threshold, determining the motion of the projection object content as a first motion,
In a projection space where there is a spatial change exceeding a threshold, the electronic device determines the motion of the projection object content to be a second motion different from the first motion. The method of any one of claims 1 to 3, wherein the one or more processors execute the one or more instructions,
Determine the operation of the projection object content based on at least one of a preset cycle, a random cycle, and event occurrence,
The occurrence of the event includes at least one of changing the position of the projection space by more than a threshold and receiving a projection object content operation control signal. The method of any one of claims 1 to 4, wherein the one or more processors execute the one or more instructions,
An electronic device that corrects at least one of color, brightness, shape, and size of the standard object content based on modeling information corresponding to the projection space. The method according to any one of claims 1 to 5, wherein the modeling information corresponding to the projection space includes projection space depth information,
The one or more processors execute the one or more instructions,
Obtain user depth information,
An electronic device that corrects the size of the standard object content using the projection space depth information and the user depth information. The electronic device according to any one of claims 1 to 6, wherein the electronic device is rotatable about at least one of the x-axis, y-axis, and z-axis,
The electronic device further includes an image sensor 106-1 and a depth sensor 106-2,
The one or more processors execute the one or more instructions,
Based on the surrounding space image acquired through the image sensor and the surrounding space depth information obtained through the depth sensor, obtain modeling information about the surrounding space,
The modeling information about the surrounding space includes color information, luminance information, and depth information according to the rotation angle for each axis of the electronic device. The method of any one of claims 1 to 7, further comprising an image sensor (106-1),
The one or more processors execute the one or more instructions to identify user feature points from the user image acquired through the image sensor,
Identify the direction the user's face faces based on the user feature points,
An electronic device that identifies a space located in the direction toward which the user's face faces as the projection space. 9. The method of any one of claims 1 to 8, wherein the projected content further comprises background content,
The electronic device wherein the one or more processors determine at least one of the color and brightness of the background content based on at least one of the color and brightness of modeling information corresponding to the projection space by executing the one or more instructions. 9. The method of any one of claims 1 to 8, wherein the projected content further comprises background content,
The electronic device wherein the one or more processors determine the brightness of the background content to a minimum value by executing the one or more instructions. In a method of operating an electronic device,
Obtaining modeling information corresponding to the projection space from modeling information about the surrounding space;
Obtaining projection object content by correcting the shape of standard object content based on modeling information corresponding to the projection space;
determining an operation of the projection object content based on modeling information corresponding to the projection space; and
A method of operating an electronic device, comprising: projecting projection content including the projection object content that performs the determined operation into the projection space. The method of claim 11, wherein the modeling information corresponding to the projection space includes projection space structure information,
Determining the operation of the projection object content includes determining the operation of the projection object content as an operation corresponding to the projection space structure information. The method of claim 12, wherein the projection spatial structure information includes spatial change information,
The step of determining the operation of the projection object content as an operation corresponding to the projection space structure information
In a projection space where there is a spatial change below a threshold, determining the motion of the projection object content as a first motion; and
A method of operating an electronic device, comprising determining an operation of the projection object content as a second operation different from the first operation in a projection space where there is a spatial change exceeding a threshold. 14. The method of any one of claims 11 to 13, wherein determining an operation of the projection object content comprises:
A step of determining an operation of the projection object content based on at least one of a preset period, a random period, and an event occurrence,
The occurrence of the event includes at least one of changing the position of the projection space by more than a threshold and receiving a projection object content operation control signal. 15. The method of any one of claims 11 to 14, wherein obtaining the projection object content comprises:
A method of operating an electronic device, comprising correcting at least one of color, brightness, shape, and size of the standard object content based on modeling information corresponding to the projection space. The method according to any one of claims 11 to 15, wherein the modeling information corresponding to the projection space includes projection space depth information,
The steps for acquiring projection object content are
Obtaining user depth information; and
A method of operating an electronic device, including correcting the size of the standard object content using the projection space depth information and the user depth information. The method of any one of claims 11 to 16, wherein the electronic device is rotatable about at least one of the x-axis, y-axis, and z-axis,
Acquiring an image of surrounding space;
Obtaining surrounding spatial depth information;
Based on the surrounding space image and the surrounding space depth information, further comprising acquiring modeling information about the surrounding space,
The modeling information about the surrounding space includes color information, luminance information, and depth information according to the rotation angle for each axis of the electronic device. The method according to any one of claims 11 to 17,
Obtaining a user image;
Identifying user feature points from the user image;
Identifying the direction the user's face faces based on the user's feature points; and
A method of operating an electronic device further comprising identifying a space located in a direction toward which the user's face faces as the projection space. 19. The method of any one of claims 11 to 18, wherein the projected content further comprises background content,
A method of operating an electronic device further comprising determining at least one of the color and brightness of the background content based on at least one of the color and brightness of modeling information corresponding to the projection space. A computer-readable recording medium on which a program capable of performing the method of any one of claims 11 to 19 is recorded by a computer.

Images