KR102214338B1 - Control method, device and program of electronic device for augmented reality display - Google Patents

Abstract

The present invention provides a control method of an electronic device. The control method comprises: a step in which a camera acquires a product image including a QR code; a step in which a processor acquires the QR code image; a step in which the processor acquires size information of a previously stored QR code from a memory; a step in which the processor acquires gaze information and the size of a product displayed on a display based on the size information of the previously stored QR code and image information of the QR code; a step in which the processor acquires information on an object corresponding to the QR code based on the gaze information and the size of the product displayed on the display; a step in which the processor acquires an area on which the object is to be displayed based on the gaze information and the size of the product displayed on the display; and a step in which the display displays the acquired object on the area.

Description

Control method, device and program of electronic device for augmented reality display

The present invention relates to a control method, an apparatus, and a program of an electronic device for displaying an augmented reality.

Virtual Reality is a technology that enables humans to experience real life in a virtual world created with computers.

On the other hand, a technology that combines virtual reality (VR) and augmented reality (AR), a technology that adds virtual information to the real world, is called mixed reality (MR). VR, AR, and MR all have something in common in that they are technologies that enable people to recognize reality by realizing a reality that does not exist. However, there is a difference in that AR is a method of adding virtual information to real reality, and in VR, all fictional situations are presented. MR mixes AR and VR to provide a mixture of real and virtual information in a real background, and a technology that can process large amounts of data is required.

In recent years, research on a method of generating and displaying a simple augmented reality with an electronic device such as a smartphone is active.

Registered Patent Publication No. 10-1397873, 2014.07.01

The problem to be solved by the present invention is to provide a method, apparatus, and program for controlling an electronic device for displaying an augmented reality.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A method of controlling an electronic device including a camera, a memory, a display, and a processor according to an aspect of the present invention for solving the above-described problems includes: obtaining, by the camera, a product image including a QR code; Obtaining, by the processor, the QR code image; Obtaining, by the processor, size information of a previously stored QR code from the memory; Obtaining, by the processor, size and gaze information of a product displayed on the display based on size information of the previously stored QR code and image information of the QR code; Acquiring, by the processor, information on an object corresponding to the QR code based on the size and gaze information of the product displayed on the display; Obtaining, by the processor, an area in which the object is to be displayed based on the size and gaze information of the product displayed on the display; And displaying, by the display, the acquired object in the area. Includes.

In this case, the control method is, when the size and gaze information of the image of the QR code is changed by more than a preset ratio according to the movement of the electronic device, the processor, the size information of the previously stored QR code and the changed QR code Acquiring the size and changed gaze information of the changed product displayed on the display based on the image information of Obtaining, by the processor, information on a changed object corresponding to the QR code and a changed area in which the object is to be displayed, based on the changed size and changed gaze information of the product displayed on the display; And displaying, by the display, the acquired changed object in the changed area. It may include.

In this case, when the object is a bonfire object, the control method may include displaying a UI for adjusting the intensity of fire power of the bonfire object in an upper center area of the display; When a user command for adjusting the intensity of the firepower of the bonfire object is input through the UI, changing and displaying the bonfire object corresponding to the user command.

In this case, when the object is a fish object, the displaying may include moving the fish object within the acquired area; And when a user command for controlling the fish object is input, moving the fish object to correspond to the user command. It may include.

In this case, the control method includes the steps of, by the camera, updating information on the QR code every preset period; And updating, by the camera, the QR code image every preset period, wherein the step of updating the information on the QR code includes an obstacle exists between the camera and the QR code, and the QR code When only a part of the QR code is photographed, restoring the image shape of the QR code based on a part of the photographed QR code; And acquiring information on a changed object corresponding to the QR code and a changed area in which the object is to be displayed, based on the image shape of the restored QR code.

In this case, the moving of the fish object may include generating voice recognition data by receiving a voice signal from a user of the electronic device; Generating a movement control block for controlling movement of the fish object and a command corresponding to the movement control block based on the speech recognition data; Controlling the movement of the fish object in response to the movement control block; Generating a command block image representing the movement control block; And displaying the command block image. It may include.

In this case, when the movement control block includes a movement speed control command, generating the movement control block and the command corresponding to the movement control block includes the movement speed information when the speech recognition data includes movement speed information. Generating a movement speed control statement corresponding to the speed information; And when movement speed information is not included in the speech recognition data, syllable spacing information is generated based on the speech recognition data, and the movement speed information is generated in response to the syllable spacing information, and the generated movement speed Generating a movement speed control statement corresponding to the information; It may include.

At this time, when the movement control block includes a movement repetition number control command, the step of generating the movement control block and the command corresponding to the movement control block includes movement repetition number information in the speech recognition data, Generating a movement repetition number control statement corresponding to the movement repetition number information; And when movement repetition number information is not included in the speech recognition data, signal strength information is generated based on the speech signal, and the movement repetition number information is generated in response to the signal strength information, and the generated movement repetition It may include; generating a movement repetition number control statement corresponding to the number information.

Other specific details of the present invention are included in the detailed description and drawings.

According to various embodiments of the present invention described above, there is a new effect that an electronic device can display an augmented reality corresponding to a QR code and control an object displayed in the augmented reality.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is an exemplary diagram illustrating an electronic device for providing augmented reality according to an embodiment of the present invention.
2 is a block diagram of an electronic device according to an embodiment of the present invention.
3 is an exemplary view for explaining the configuration of a QR code according to an embodiment of the present invention.
4A to 4D are exemplary diagrams illustrating a method of displaying an augmented reality according to an exemplary embodiment of the present invention.
5A to 5D are exemplary diagrams illustrating a method of displaying an augmented reality according to another embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

The term "unit" or "module" used in the specification refers to a hardware component such as software, FPGA or ASIC, and the "unit" or "module" performs certain roles. However, "unit" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Thus, as an example, "sub" or "module" refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, It includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. Components and functions provided within "sub" or "module" may be combined into a smaller number of components and "sub" or "modules" or into additional components and "sub" or "modules". Can be further separated.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc., as shown in the figure It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, if a component shown in a drawing is turned over, a component described as "below" or "beneath" of another component will be placed "above" the other component. I can. Accordingly, the exemplary term “below” may include both directions below and above. Components may be oriented in other directions, and thus spatially relative terms may be interpreted according to orientation.

In the present specification, a computer refers to all kinds of hardware devices including at least one processor, and may be understood as encompassing a software configuration operating in a corresponding hardware device according to embodiments. For example, the computer may be understood as including all of a smartphone, a tablet PC, a desktop, a laptop, and a user client and an application running on each device, but is not limited thereto.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings based on FIGS. 1 to 5D.

Each of the steps described herein is described as being performed by a computer, but the subject of each step is not limited thereto, and at least some of the steps may be performed by different devices according to embodiments.

1, the electronic device 100 may photograph a product including a QR code. The electronic device 100 may display a background image acquired through the camera 110 on the display 130 and display an object corresponding to the QR code information by superimposing it on the display 130.

2 is a block diagram of an electronic device according to an embodiment of the present invention.

The electronic device 100 according to the present invention includes a camera 110, a processor 120, a display 130, an input unit 140, a microphone 150, a speaker 160, a memory 170, and a communication unit 180. Can contain

The camera 110 may take a background image. Specifically, the camera 110 may photograph the QR code included in the background image and display the background image and the object by overlapping.

The camera 110 is a component capable of photographing an image as described above. The camera 110 may include a lens, a shutter, an aperture, an image sensor, an analog front end (AFE), and a timing generator (TG).

Specifically, a lens (not shown) has a configuration in which light reflected on a subject is incident, and may include at least one of a zoom lens and a focus lens. The shutter (not shown) adjusts the time when light enters the image capturing device 100. The amount of light accumulated in the exposed pixels of the image sensor is determined according to the shutter speed. The diaphragm (not shown) is a component that adjusts the amount of light incident into the image capturing apparatus 100 through the lens. An image sensor (not shown) is a configuration in which an image of a subject passing through a lens is formed.

The image processing unit 170 may process raw image data photographed by the camera 110 to produce YCbCr data. In addition, it is possible to determine the image black level and adjust the sensitivity ratio for each color. In addition, the image processing unit 170 may adjust white balance, perform gamma correction, color interpolation, color correction, and resolution conversion.

The processor 120 controls the overall operation of the electronic device 100.

The processor 120 may include a RAM, a ROM, a main CPU, first to n interfaces, and a bus. In this case, RAM, ROM, main CPU, first to n interfaces, and the like may be connected to each other through a bus.

The ROM stores an instruction set for booting the system. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory 170 to RAM according to the instruction stored in the ROM, and executes the O/S to boot the system. When booting is completed, the main CPU copies various application programs stored in the memory 170 to RAM, and executes the application programs copied to the RAM to perform various operations.

The main CPU accesses the memory 170 and performs booting using the O/S stored in the memory 170. In addition, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory 170.

The first to nth interfaces are connected to the various components described above. One of the interfaces may be a network interface that is connected to an external device through a network.

The display 130 may be implemented with various types of display panels. For example, the display panel is LCD (Liquid Crystal Display), OLED (Organic Light Emitting Diodes), AM-OLED (Active-Matrix Organic Light-Emitting Diode), LcoS (Liquid Crystal on Silicon) or DLP (Digital Light Processing) It can be implemented with various display technologies such as. In addition, the display 130 may be coupled to at least one of a front area, a side area, and a rear area of the electronic device 100 in the form of a flexible display.

The input unit 140 may receive a user command. The input unit 140 may be configured in various forms. For example, the input unit 140 may be implemented as a touch screen in combination with the display 130 and a touch sensing unit (not shown). However, it is not limited to this configuration, and the input unit 140 may include a button or may be composed of an external remote control, may be composed of a microphone 150 for voice input, and motion input by combining with the camera 110 You can also do

In particular, the input unit 140 may receive a user command for editing a product image. In this case, the input user command may be a long press command, a double tap command, a swipe command, or the like.

The memory 170 may store an O/S (Operating System) for driving the electronic device 100. Also, the memory 170 may store various software programs or applications for operating the electronic device 100 according to various embodiments of the present disclosure. The memory 170 may store various information such as various data input, set, or generated during execution of a program or application.

In addition, the memory 170 may include various software modules for operating the electronic device 100 according to various embodiments of the present disclosure, and the processor 120 executes various software modules stored in the memory 170 An operation of the electronic device 100 according to various embodiments of the present disclosure may be performed.

In addition, the memory 170 may store spatial images captured by the camera 110 and various images received from the outside. To this end, the memory 150 may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

The communication unit 180 may communicate with an external device. In particular, the communication unit 180 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, an NFC chip, and a wireless communication chip. At this time, the WiFi chip, the Bluetooth chip, and the NFC chip perform communication using a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. When using a Wi-Fi chip or a Bluetooth chip, various types of connection information such as an SSID and a session key may be transmitted and received first, and then various types of information may be transmitted and received after a communication connection using them. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE). In particular, the communication unit 180 May receive various information from an external device (eg, a content server that provides a product image). For example, the communication unit 180 may receive various indoor images, product information, and product images from an external device, and store the received information in the memory 170.

3 is an exemplary view for explaining the configuration of a QR code according to an embodiment of the present invention.

One QR Code is divided into a function pattern and an encoding region, and the detailed function pattern consists of a finder pattern, alignment pattern, and timing pattern, and the encoding area includes format information, version information, and data. It can be composed of elements such as and error correction codewords.

At this time, function patterns can always occupy the same place and space in the QR Code. The functional patterns help high-speed recognition and improvement of recognition rate.

In addition, detailed information of each component is as follows. First, the finder pattern can be fixedly positioned in three directions. Second, there may be two dotted timing patterns connecting three finder patterns. Third, the format information may be a square box surrounding the finder pattern. Fourth, as for the version information, version information may be located on the left and upper sides of the direction pattern from version 7 or higher, and may be located at a position where the alignment pattern is fixed from version 2 or higher. However, it goes without saying that the components of the QR Code can be configured somewhat differently depending on the version.

Hereinafter, the operation of the present invention will be described in detail based on the configuration of the electronic device 100 described above.

First, the camera 110 may acquire a product image including a QR code. Thereafter, the processor 120 may acquire a QR code image. Thereafter, the processor 120 may obtain size information of the previously stored QR code from the memory 170.

Thereafter, the processor 120 may acquire size and gaze information (or tilt information) of a product displayed on the display 130 based on the previously stored size information of the QR code and image information of the QR code.

Thereafter, the processor 120 may obtain information on the object corresponding to the QR code based on the size and gaze information (or tilt information) of the product displayed on the display 130.

Specifically, even if the same product is photographed, the shape of the product displayed on the display may be different depending on the photographing angle. For example, as shown in FIGS. 4A, 4B, and 5A, 5B, a case where the electronic device 100 and the product are photographed on a parallel plane and shown in FIGS. 4C, 4D, 5C, and 5D As described above, when the electronic device 100 and the product are photographed on a non-parallel (ie, oblique) plane, the background image displayed on the electronic device 100 may be different.

Accordingly, the electronic device 100 pre-stores the size information of the QR code (for example, a square of 3 cm in width and 3 cm in height) in the memory 170, and the size of the QR code on the image captured by the camera 110 Based on the information, the size and inclination of the object, which will be described later, can be determined. For example, the size information of the QR code on the captured image is a square 2cm in width and 2cm in height, and the electronic device 100 determines the size of the object. It can be enlarged 1.5 times larger than a preset size (that is, a size corresponding to a square QR code of 3 cm and 3 cm in height). As another example, when the size information of the QR code on the photographed image is a square 3cm in width and 2cm in height, the electronic device 100 moves the object at an inclination corresponding to the QR code 3cm in length and 3cm in height. Can be converted.

Thereafter, the processor 120 may acquire an area in which the object is to be displayed based on the size and gaze information (or tilt information) of the product displayed on the display 130.

That is, the electronic device 100 may determine the size and inclination of the object based on the size information of the QR code on the image captured by the camera 110 as well as determine the area in which the object is to be displayed.

That is, as shown in FIG. 5B, the area when the electronic device 100 and the QR code are located on a parallel plane is a plane in which the electronic device 100 and the QR code are parallel as shown in FIG. 5D. It may be a larger area than an area not located on the image.

Through the above-described process, when the product includes a fish object as shown in FIGS. 5A to 5D, the electronic device 100 may allow the fish to be located only in the water area.

Thereafter, the display 130 may display the acquired object on the area.

On the other hand, when the size and gaze information (or tilt information) of the image of the QR code is changed by more than a preset ratio according to the movement of the electronic device, the processor 120, the size information of the previously stored QR code and the image of the changed QR code Based on the information, the size of the changed product and changed gaze information (or tilt information) displayed on the display 130 may be obtained.

Thereafter, the processor 120 acquires information on the changed object corresponding to the QR code and the changed area in which the object is to be displayed, based on the changed size and changed gaze information (or tilt information) of the product displayed on the display 130 can do.

Thereafter, the display 130 may display the acquired changed object in the changed area.

That is, when the electronic device 100 is implemented as a smart phone, in general, a smart phone that photographs a QR code will move more freely than when it exists in a fixed location. For a slight movement such as hand shake, the electronic device 100 does not need to change an object or an area in which the object is displayed, but in case of a large movement, the need to change the object or an area in which the object is displayed increases.

Accordingly, when the QR code captured by the camera 100 is changed by a preset ratio (eg, 10%) or more, the electronic device 100 may change the size and slope of the object, and the area in which the object is to be displayed.

On the other hand, according to an embodiment of the present invention, as shown in FIGS. 4A to 4D, when the object is a bonfire object, the electronic device 100 is in the upper central area of the display 130, and the fire power of the bonfire object is You can display a UI for adjusting

According to another embodiment of the present invention, when a user command for adjusting the intensity of firepower of a bonfire object is input through the UI, the electronic device 100 may change and display the bonfire object corresponding to the user command.

Meanwhile, as illustrated in FIGS. 5A to 5D, when the object is a fish object, the electronic device 100 may move the fish object within the acquired area.

Thereafter, when a user command for controlling the fish object is input, the electronic device 100 may move the fish object corresponding to the user command.

Meanwhile, the camera 110 may update information on the QR code (ie, the type of the object to be displayed and the area information in which the object is to be displayed) every preset period.

That is, if the information on the QR code is not updated every preset period, the electronic device 100 continues to display the object even if the QR code disappears on the display 130.

Accordingly, the electronic device 100 may determine whether the QR code is photographed by the camera 110 every preset period, and if the QR code is not photographed by the camera 110, the object display may be stopped.

Thereafter, the camera 110 may update the QR code image every preset period.

Specifically, even when the QR code is photographed by the camera 110, there is a need to change the size and inclination of the object, and the area in which the object is to be displayed, as the position of the electronic device 100 is changed. Accordingly, the electronic device 100 may update the QR code image (including size information) as well as the QR code information.

On the other hand, in the step of updating the information on the QR code, when there is an obstacle between the camera 110 and the QR code, and only part of the QR code is photographed, the image shape of the QR code is determined based on a part of the photographed QR code. Can be restored.

Specifically, when displaying an object for the first time, information of the QR code of the electronic device 100 must be completely received, but after the object is displayed, only whether the electronic device 100 is photographing the QR code is important. In this case, there may be a case where an obstacle exists between the camera 110 and the QR code, so that part of the QR code disappears from the display 130 due to the obstacle.

However, even if part of the QR code disappears on the display 130 due to an obstacle, the electronic device 100 continuously displays the object may be an alternative for constructing a smooth augmented reality.

Accordingly, even when a part of the QR code disappears from the display 130 due to an obstacle, the electronic device 100 may acquire the image shape of the QR code based on a partial area of the QR code obtained by the display 130. . That is, in this case, the image shape of the QR code acquired by the electronic device 100 may be information about the size and inclination of the QR code, not information obtained by decoding the QR code.

Thereafter, based on the image shape of the restored QR code, information on a changed object corresponding to the QR code and a changed area in which the object is to be displayed may be obtained.

Meanwhile, according to various embodiments of the present disclosure, the fish object illustrated in FIGS. 5A to 5D may be moved according to a user command. In this case, the user command may be a voice command, but is not limited thereto.

For example, the user command may be a command for dragging and moving a fish displayed on the display 130. Alternatively, it goes without saying that the user command may be a hover command, a double tap input, or the like.

Hereinafter, a process of moving a fish object based on a voice command will be described.

The electronic device 100 may receive a voice signal from a user and generate voice recognition data. In one embodiment, the processor 120 displays a voice command input button from the user on the display 130, and activates the microphone 150 to receive a voice command when a command through the voice command input button is input from the user. I can make it.

Here, the speech recognition data may be data generated based on a result of preprocessing a speech signal by the processor 120, extracting a speech characteristic from the preprocessed speech signal, comparing the extracted speech characteristic with a speech model, and the comparison result. have. For example, the voice recognition data may be text corresponding to a voice command.

In this case, the type of speech recognition method used by the processor 120 to generate speech recognition data corresponding to a speech command is not limited as long as the speech recognition data is generated.

In an embodiment, the processor 120 may control the display 130 to display a speech recognition data image representing the recognized speech recognition data.

In another embodiment, the processor 120 may control the display 130 to display a confirmation input area through which a user can input whether or not the recognized voice recognition data corresponds to a voice command.

The processor 120 may perform a process of generating a movement control block when a confirmation input indicating that the recognized speech recognition data corresponds to a speech command is input. Conversely, the processor 120 may perform a process of re-inputting the voice command when an unconfirmed input indicating that the recognized voice recognition data does not correspond to the voice command is input.

Thereafter, the processor 120 may generate a movement control block for controlling the movement of the fish object and a statement corresponding to the movement control block based on the speech recognition data, and control the movement of the fish object in response to the movement control block. .

Thereafter, the processor 120 may generate a command block image representing the movement control block and control the display 130 to display the command block image.

In this case, the movement control block may include at least one of a movement type control command, a movement speed control command, and a movement repetition number control command corresponding to the set programming language type.

In addition, the movement type control statement includes a movement type control command and movement type control variable corresponding to the set programming language type, and the movement speed control statement includes a movement speed control command and movement speed control variable corresponding to the set programming language type. And, the movement repetition number control command may include a movement repetition number control command and a movement repetition number control variable corresponding to a set programming language type.

The processor 120 may search for movement type information from the voice recognition data and, when the movement type information is searched, may generate a movement type control command corresponding to the searched movement type information.

For example, when the voice recognition data is "move to the right quickly", the movement type information may be "right", and the movement type control command corresponding to the movement type information may be "#move right". To this end, the text corresponding to the movement type information may be matched with a movement type control statement for each programming language type and stored in the memory 170.

Meanwhile, when the movement control block includes a movement speed control command, the processor 120 may generate a movement speed control command corresponding to the movement speed information when the movement speed information is included in the speech recognition data.

On the other hand, when the movement speed information is not included in the speech recognition data, the processor 120 generates syllable interval information based on the speech recognition data, generates movement speed information corresponding to the syllable interval information, and generates A movement speed control statement corresponding to movement speed information can be generated.

Specifically, the processor 120 may search for movement speed information from speech recognition data, and when the movement speed information is searched, may generate a movement speed control statement corresponding to the searched movement speed information. For example, when the voice recognition data is "move to the right quickly", the movement speed information may be "fast", and the movement type control command corresponding to the movement speed information may be "#speed up".

To this end, a text corresponding to the movement speed information may be matched with a movement speed control statement for each programming language type and stored in the memory 170.

Conversely, if the movement speed information is not retrieved from the speech recognition data, the processor 120 generates syllable interval information based on the speech recognition data, generates movement speed information corresponding to the syllable interval information, and generates movement speed information. You can create a movement speed control statement corresponding to.

Specifically, the processor 120 calculates a first time difference between the end time of one syllable and the start time of the next syllable among the syllables included in the speech recognition data, and the average time difference between the plurality of first time differences Can be generated as syllable spacing information.

Thereafter, the processor 120 may calculate a time difference ratio of the average time difference indicated by the syllable interval information relative to the reference time difference, and apply the time difference ratio to the basic movement speed to generate movement speed information.

For example, when the average time difference is "12msec", the reference time difference is "10msec", and the basic movement speed is "1x", the processor 120 calculates the time difference ratio as "120%", and the basic movement speed By applying a time difference ratio of "120%" to "1x speed", movement speed information can be generated as "1.2x speed".

Meanwhile, when the movement control block includes the movement repetition number control statement, the processor 120 generates a movement repetition number control statement corresponding to the movement repetition number information when the movement repetition number information is included in the speech recognition data. I can.

On the other hand, when the movement repetition number information is not included in the speech recognition data, the processor 120 generates signal strength information based on the speech signal, generates movement repetition number information corresponding to the signal strength information, and generates A movement repetition number control statement corresponding to the movement repetition number information can be generated.

Specifically, on the other hand, the processor 120 may search for movement repetition number information from the speech recognition data, and when the movement repetition number information is searched, may generate a movement repetition number control statement corresponding to the searched movement repetition number information. For example, when the voice recognition data is "Twice the number of times," the movement repetition number information may be "twice," and the movement type control command corresponding to the movement repetition number information may be "#two".

To this end, a text corresponding to the movement repetition number information may be matched with a movement repetition number control statement for each programming language type and stored in the memory 170.

Conversely, if the movement repetition number information is not retrieved from the speech recognition data, the processor 120 generates signal strength information based on the voice signal, generates movement repetition number information in response to the signal strength information, and generates the movement repetition number. It is possible to generate a movement repetition control statement corresponding to the information.

Specifically, the processor 120 may generate the maximum signal strength as signal strength information among signal strengths of the voice signal over time.

Thereafter, the processor 120 may calculate a signal intensity ratio of the maximum signal intensity indicated by the signal intensity information to the reference signal intensity, and apply the signal intensity ratio to the basic movement repetition number to generate the movement repetition number information.

For example, when the maximum signal strength is "200DB", the reference signal strength is "100DB", and the number of basic movement repetitions is "one time", the processor 120 calculates the signal strength ratio as "200%", By applying the signal strength ratio "200%" to the basic number of movement repetitions "once", information on the number of movement repetitions may be generated as "2 times". At this time, the movement repetition number information may be rounded up and generated as a natural number.

Meanwhile, the processor 120 may generate user characteristic information of a user based on the voice signal, and set a programming language type of the movement control block in response to the user characteristic information.

Here, the user characteristic information may include user gender information and user age information.

Specifically, the processor 120 calculates the average frequency from the frequency of the voice signal over time, and reads the user age information of the user corresponding to the average frequency from the frequency-user characteristic table in which user age information is mapped for each voice frequency. I can.

On the other hand, according to various embodiments of the present invention, it goes without saying that the movement control block may include various commands. In an embodiment, the processor 120 may control the object to perform a specific operation according to the frequency of the input voice. For example, when the frequency of the input voice is out of a preset range (for example, 60 Hz to 80 Hz), the processor 120 may control the object so that the fish object jumps out of the water. Specifically, when the input frequency is less than 60 Hertz, the processor 120 may control the object to perform an operation in which a fish object having a predetermined size or larger among the plurality of fish objects displayed on the display 130 jumps out of the water. Alternatively, when the input frequency exceeds 80 hertz, the processor 120 may control the object so that a fish object having a predetermined size or less among the plurality of fish objects displayed on the display 130 jumps out of the water.

In another embodiment, it goes without saying that the processor 120 may perform an operation in which the fish object jumps higher as the frequency of the input voice increases.

Meanwhile, according to another embodiment of the present invention, the electronic device 100 may display an object differently according to the focal length of the background image displayed on the display 130. For example, when the focus of the camera 110 is focused on the product displayed on the display 130, the processor 120 clearly displays the object, and the camera 110 is placed in an area other than the product displayed on the display 130. When the focus of is focused, the processor 120 may blur the object.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Components of the present invention may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a computer that is hardware. Components of the present invention may be implemented as software programming or software elements, and similarly, embodiments include various algorithms implemented with a combination of data structures, processes, routines or other programming elements, including C, C++ , Java, assembler, or the like may be implemented in a programming or scripting language. Functional aspects can be implemented with an algorithm running on one or more processors.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

100: electronic device

Claims (10)

In the control method of an electronic device including a camera, a memory, a display and a processor,
Obtaining, by the camera, a product image including a QR code;
Obtaining, by the processor, the QR code image;
Obtaining, by the processor, size information of a previously stored QR code from the memory;
Obtaining, by the processor, size information and gaze information of a product displayed on the display based on size information of the previously stored QR code and image information of the QR code;
Obtaining, by the processor, information on an object corresponding to the QR code based on the size and gaze information of the product displayed on the display;
Obtaining, by the processor, an area in which the object is to be displayed based on the size and gaze information of the product displayed on the display; And
Displaying, by the display, the acquired object in the area; Including,
If the object is a fish object,
The displaying step,
Moving the fish object within the acquired area; And
When a user command for controlling the fish object is input, moving the fish object to correspond to the user command; Including,
The step of moving the fish object,
Generating voice recognition data by receiving a voice signal from a user of the electronic device;
Generating a movement control block for controlling movement of the fish object and a statement corresponding to the movement control block based on the speech recognition data;
Controlling the movement of the fish object in response to the movement control block;
Generating a command block image representing the movement control block; And
Displaying the command block image; Control method comprising a. The method of claim 1,
The control method,
When the size and gaze information of the image of the QR code is changed by more than a preset ratio according to the movement of the electronic device, the processor, based on the size information of the previously stored QR code and image information of the changed QR code, Acquiring the size of the changed product and changed gaze information displayed on the display;
Obtaining, by the processor, information on a changed object corresponding to the QR code and a changed area in which the object is to be displayed, based on the changed size and changed gaze information of the product displayed on the display; And
Displaying, by the display, the acquired changed object in the changed area; Control method comprising a. The method of claim 1,
If the object is a bonfire object,
The control method,
Displaying a UI for adjusting the intensity of heat of the bonfire object in the upper center area of the display;
When a user command for adjusting the intensity of the firepower of the bonfire object is input through the UI, changing and displaying the bonfire object corresponding to the user command. delete The method of claim 2,
The control method,
Updating, by the camera, information on the QR code every preset period; And
Including, the camera, updating the QR code image every preset period,
Updating the information on the QR code,
When an obstacle exists between the camera and the QR code and only a part of the QR code is photographed, restoring an image shape of the QR code based on a part of the QR code; And
And acquiring information on a changed object corresponding to the QR code and a changed area in which the object is to be displayed based on the image shape of the restored QR code. delete The method of claim 1,
When the movement control block includes a movement speed control command, generating a command corresponding to the movement control block and the movement control block,
Generating a movement speed control statement corresponding to the movement speed information when movement speed information is included in the speech recognition data; And
When movement speed information is not included in the speech recognition data, syllable spacing information is generated based on the speech recognition data, and the movement speed information is generated in response to the syllable spacing information, and the generated movement speed information Generating a movement speed control statement corresponding to a; Control method comprising a. The method of claim 1,
When the movement control block includes a movement repetition number control statement, generating a command corresponding to the movement control block and the movement control block,
Generating a movement repetition number control statement corresponding to the movement repetition number information when the movement repetition number information is included in the speech recognition data; And
When the speech recognition data does not include movement repetition number information, signal strength information is generated based on the voice signal, and the movement repetition number information is generated in response to the signal strength information, and the generated movement repetition number The control method comprising; generating a movement repetition number control statement corresponding to the information. A memory for storing one or more instructions; And
And a processor that executes the one or more instructions stored in the memory,
The processor executes the one or more instructions,
An apparatus for performing the method of claim 1. A computer program combined with a computer as hardware and stored in a recording medium readable by a computer to perform the method of claim 1.

Images