WO2021107186A1

WO2021107186A1 - Smart wall and control method therefor

Info

Publication number: WO2021107186A1
Application number: PCT/KR2019/016539
Authority: WO
Inventors: 임철민; 최주영; 홍지표
Original assignee: 엘지전자 주식회사
Priority date: 2019-11-28
Filing date: 2019-11-28
Publication date: 2021-06-03
Also published as: KR20220108027A

Abstract

The present invention relates to a smart wall and a control method therefor, and the substance thereof is to: capture, via a first camera, at least one first image including a user and an object in front of a screen; capture, via a second camera, at least one second image including the user and the object; recognize a motion of the user on the basis of the at least one first image and the at least one second image; recognize the object on the basis of the at least one first image and the at least one second image; determine an intention of the user on the basis of the recognized motion of the user and the recognized object; and control a multimedia device on the basis of the determined intention of the user.

Description

Smart wall and its control method

The present invention relates to a smart wall and a control method thereof, and more particularly, to a smart wall that determines a user's intention based on an image captured by a camera mounted on the smart wall, and controls a multimedia device based on the determined intention; It's about how to control it.

Recently, as the infrastructure that connects all media devices and home appliances in the home beyond smartphones to the Internet has been established, the smart home is emerging as a keyword of a new smart ecosystem. In particular, the experiences and values of users already accumulated through smartphones, which were personal media devices, began to expand to other devices.

As such, various attempts have been made to establish a home network system in order for media devices and home appliances to interoperate and operate, and the number of electronic devices in the home increases, and it is cumbersome to connect them one by one. Therefore, in order to reduce the number of electronic devices, the integration work of tying the TV, air purifier, and speaker into one is in progress.

For example, a smart TV, a multimedia device such as a DVD player, an air purifier, a console game machine, etc. may be installed in the mounting space of the smart wall.

However, in the case of the prior art, since there is no way to know the intention of the user, when the content of the multimedia device is reproduced, the user has to manually set the environment of the multimedia device, so that the user feels inconvenient.

An embodiment of the present invention analyzes an image captured by a camera to recognize a user's motion and object, determine the user's intention based on the recognized user's motion and the recognized object, and match the determined user's intention An object of the present invention is to provide a smart wall for controlling a multimedia device and a method for controlling the same.

Another embodiment of the present invention aims to provide a smart wall and a control method therefor that, when there are a plurality of users, identify each user and individually determine the user's intention based on the identified user.

Another embodiment of the present invention aims to provide a smart wall for adjusting the volume of a sound output from a speaker in response to a specific action when a user performs a specific action, and a method for controlling the same.

The technical problems to be achieved in the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. .

According to an aspect of the present invention, a smart wall includes a display for displaying a graphic image; a screen that covers the front surface of the display and is rolled on the display to adjust a screen size of the display; a driving roller positioned on the screen and wound around the screen; a motor providing rotational force to the driving roller; a screen sensing unit for sensing the size of the first area of the screen; a first camera for capturing at least one first image including a user and an object in front of the screen; a second camera configured to capture at least one second image including the user and the object; control the driving of the motor based on the size of the first area sensed by the screen sensing unit, recognize a user's motion based on the at least one first image and the at least one second image, and Recognizing the object based on the at least one first image and the at least one second image, determining the user's intention based on the recognized user's motion and the recognized object, and determining the determined user's intention and a control unit for controlling the multimedia device based on the multi-media device.

According to another aspect of the present invention, there is provided a method for controlling a smart wall, the method comprising: capturing at least one first image including a user and an object in front of a screen with a first camera; capturing with a second camera that captures at least one second image including the user and the object; recognizing a user's motion based on at least one first image and the at least one second image; recognizing the object based on at least one first image and the at least one second image; determining a user's intention based on the recognized user's motion and the recognized object; and controlling the multimedia device based on the determined intention of the user.

According to an embodiment of the present invention, the user's motion and object are recognized by analyzing the image captured by the camera, the user's intention is determined based on the recognized user's motion and the recognized object, and the determined user's intention Since it is possible to control a multimedia device to conform to it, it is possible to control the multimedia device only with the user's natural motion without a separate remote control operation, thereby improving user convenience.

According to another embodiment of the present invention, when there are a plurality of users, the user can be identified for each individual, and the user's intention can be determined for each individual based on the identified user, so that the environment of the multimedia device can be set for each individual. can improve

According to another embodiment of the present invention, when the user performs a specific action, the volume of the sound output by the speaker can be adjusted in response to the specific action to cope with a specific situation, thereby improving user convenience.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a view showing an installation example of the smart wall of the present invention.

2 is an exploded perspective view of a smart wall according to an embodiment of the present invention.

3 is a block diagram of a smart wall according to an embodiment of the present invention.

4 is a diagram illustrating the configuration of a frame of a smart wall according to an embodiment of the present invention.

5 shows examples of various frame structures according to the combination of frame modules of the present invention.

6 is a diagram for explaining a frame module of a smart wall according to an embodiment of the present invention.

7 is a view for explaining a method of installing the frame structure of the smart wall according to an embodiment of the present invention.

8 is a view for explaining a basket of a smart wall according to an embodiment of the present invention.

9 is a diagram illustrating a change in a screen according to a size at which the display unit of the smart wall is exposed to the front according to an embodiment of the present invention.

10 is a diagram illustrating an example of a screen sensing unit of a smart wall.

FIG. 11 is a view for explaining an operation of the screen sensor according to a change in the size of the first area of the screen sensor of FIG. 10 .

12 is a diagram illustrating a modified example of the screen sensing unit of FIG. 10 .

13 is a diagram illustrating another example of a screen sensing unit of a smart wall.

14 is a diagram illustrating a specific structure of the screen sensing unit of FIG. 13 .

15 is a diagram illustrating a configuration of a smart wall according to an embodiment of the present invention.

16 is a flowchart illustrating a method for controlling a smart wall according to an embodiment of the present invention.

17 is a diagram illustrating a hardware configuration of a smart wall according to an embodiment of the present invention.

18 is a diagram illustrating a hardware configuration of a smart wall according to an embodiment of the present invention.

19 is a diagram illustrating a line view, a half view, and a full view of a smart wall according to an embodiment of the present invention.

20 is a diagram illustrating user recognition, user motion recognition, and object recognition with a camera mounted on a smart wall, according to an embodiment of the present invention.

21 is a diagram illustrating setting of an environment of a multimedia device based on a user's intention according to an embodiment of the present invention.

22 is a diagram illustrating setting of an environment of a multimedia device based on a user's intention according to an embodiment of the present invention.

23 is a diagram illustrating setting of an environment of a multimedia device based on a user's intention according to an embodiment of the present invention.

24 is a diagram illustrating adjusting a volume of a speaker when a user makes a phone call, according to an embodiment of the present invention.

25 is a diagram illustrating pause of content reproduced by a multimedia device when a user disappears from a camera angle of view, according to an embodiment of the present invention.

26 is a diagram illustrating determination of a user's intention using a reinforcement learning model, according to an embodiment of the present invention.

27 is a diagram illustrating recognition of a user's motion based on a current image and a previous image, according to an embodiment of the present invention.

28 is a diagram illustrating photographing a body point with a camera and recognizing a user's motion based on the body point, according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted.

In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as "comprises" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

1 is a view showing a wall-type frame of the smart wall 100 of the present invention. The present invention relates to a smart wall 100 in which electronic devices used at home, such as media devices and home appliances, are integrated into the wall so as not to be exposed to the exterior. Various electronic devices such as displays, intercoms, speakers, air purifiers, humidifiers, lighting, digital watches, routers, and set-top boxes are mounted on the wall to provide a space total solution that minimizes the exposure of electronic devices in the space.

Recently, as the thickness of the display device 161 becomes thinner, the thickness of the smart wall 100 of the present invention can be formed as thin as 15 to 20 cm, thereby preventing the narrowing of the indoor space during installation. Rather, the storage furniture and home appliances 160 may be embedded in the smart wall 100 to provide a neat appearance. If necessary, a part of the wall may be selectively pulled out so that the necessary home appliances may protrude, and the space in which the home appliances are not buried may be utilized as a mounting space 114 such as a drawer.

A TV can be thinned in recent years to realize a thin display unit. In fact, the TV itself serves as a cover for the frame and can be combined with the frame to form a front appearance. An opening 153 may be formed in the art wall 151 to expose the display unit of the display device 161 , and a cover for opening and closing the opening 153 may be further provided.

Since the display unit using OLED is curved, it is exposed to the outside only when in use, and when not in use, it can be rolled up and stored inside the wall, thereby providing a neat screen in appearance.

As for the projector, a short throw projector has recently been developed, and the projector can be mounted on a wall and drawn out in the form of a drawer and outputted to the screen to realize a large screen. In this case, when not in use, the display unit is not visible on the outside of the smart wall 100 to provide a cleaner appearance.

Figure 2 is an exploded view of the smart wall 100 of the present invention, Figure 3 is a block diagram of the smart wall 100 of the present invention, the inside of the front art wall 150 is a plurality of shape steel 115 (shape steel) ) can be located in a frame structure arranged in a grid. The frame structure 110 is a rigid member and can be supported so as not to collapse even when an impact is applied to the wall. By attaching a panel to the front surface of the frame structure 110 , the electronic product 160 mounted on the frame structure 110 may be hidden and configured to be viewed as a wall. The front wall covered with such a panel is called an art wall 150 .

The art wall 150 can be changed differently according to the taste of the consumer. It can also be easily changed after installation to change the atmosphere of the house.

The section steel 115 arranged in a lattice form may partition the mounting space 114 , and the electronic product 160 may be seated in each mounting space 114 . Each mounting space 114 may have the same size or multiple sizes, so that the mounted electronic product 160 may be modularized and disposed at any position within the frame structure 110 .

In addition to the display device 161 described above, an acoustic device 162 such as a speaker may be mounted in the mounting space 114 , and a plurality of acoustic devices 162 may be configured according to the number of channels. The art wall 152 covering the sound device 162 may be configured in a mesh shape to output sound or selectively open and close the opening through which sound is output to minimize the opening exposed to the art wall 152 . It is also possible to prevent water from entering through the opening by using a waterproof mesh or the like.

Seasonal home appliances, such as a dehumidifier 163, a humidifier 165, an air purifier 164 and air conditioner, may be provided with a heater 167 such as a stove. When these seasonal household appliances are not in use, if they are left exposed to the outside, dust accumulates and looks unsightly, so it is common to put a cover or put them in a warehouse. However, if buried in the smart wall 100 as in the present invention, the art wall 150 can cover it so that it is not exposed to the outside even if it is left as it is even when not in use. That is, a separate storage space is provided for seasonal home appliances. no need to do

The dehumidifier 163 or the humidifier 165 may implement a space for collecting/storing water to remove the collected water or to fill the water in a drawer type. An opening 154 through which air passes is required for water to be discharged or to collect surrounding moisture.

Such an opening 154 may use the gap between the art wall 150 and the frame structure 110, even if it is not necessarily formed in the art wall 150.

The air purifier 164 may include an openable and openable cover to periodically replace the filter.

Each of the electronic products may be individually controlled, but may further include a controller 180 for integrally controlling the electronic products mounted on the smart wall. The controller may have its own memory 185 to store the collected data.

The dehumidifier, humidifier and air purifier 164 may be installed at the lower part of the smart wall 100 so that the user can easily operate it. In addition, if a set-top box or WIFI router 175 is mounted, a home network system can be implemented. can Through the WIFI router 175, not only the functional unit mounted on the smart wall 100, but also other electronic devices such as a computer or a mobile terminal located outside the smart wall 100 can be connected and interoperable, and the smart wall 100 It may include a controller 180 for controlling each electronic product 160 mounted on the.

By arranging an LED or a small display panel on the upper portion of the smart wall 100, simple information such as weather and time event information may be provided to the user, or may be used as lighting. An interior effect can be added by arranging lighting on the smart wall 100 itself.

The router 175 is an intermediate device through which the router organizes and guides the way when different communication methods or signals are used. In addition, the functional unit mounted on the smart wall 100 can be controlled by connecting it through the WIFI router 175 without necessarily connecting it by wire.

In addition, since the router 175 serves to connect the external network and the internal network at the same time, the user can access the home network as well as the electronic product 160 mounted on the smart wall 100 through the router 175 connected to the external network from the outside. It can control other home appliances and media devices connected to it.

In addition, the controller 180 may control the electronic devices of the smart wall 100 in conjunction with the mobile device. That is, when a mobile device is mounted on the smart wall 100, media images and music in the mobile device are output, or when a video call is made through the mobile device, information from the mobile device is transmitted through Bluetooth or Wi-Fi to the smart wall 100. ) may output the image of the other party to the display device 161 .

In addition, a wall pad connected to a camera or method system at the entrance may be provided in the mounting space 114 of the smart wall 100 . The wall pad embedded in the existing load-bearing wall was difficult to repair and replace and difficult to change in case of failure, but it has the advantage of easy maintenance when it is configured in a unit form and provided in the smart wall 100 .

The smart wall 100 may be installed in the living room as well as the room. By configuring the wall between the rooms as the smart wall 100, the smart wall 100 can be used in both directions in a neighboring room. For example, if one air conditioner or air purifier is allowed to flow in both directions, it can be used in both spaces. Instead of the TV and speaker 162 used in the living room, a monitor and a PC may be mounted and used according to the purpose of the room. Alternatively, different functional units, such as an electronic board, may be mounted and used according to the age of the child.

When installed in an office, a smart wall 100 equipped with functional units such as an electronic board, an air purifier, a video conference system, and a PC can be used. Since the electronic product 160 mounted on the smart wall 100 is covered by the art wall 150, a separate exterior case is not required.

When the smart wall 100 is manufactured and installed in the factory according to the user's installation location, the same module can be manufactured and supplied on a large scale when constructing a hotel/resort or apartment, thereby reducing the defect rate and producing at a low cost.

However, the assembled smart wall 100 is difficult to transport, and when the building is completed, it is difficult to bring it into the building. Therefore, when the smart wall 100 is installed when moving to an existing building or interior, the smart wall 100 can be implemented by installing a frame in the building and inserting a unit module. In this case, for convenience of installation and standardization, the frame structure 110 may be implemented using a plurality of module frames.

4 shows a coupling method of the

frame modules

111, 112, and 113 of the smart wall 100 according to an embodiment of the present invention. In order to mount each component,

frame modules

111 , 112 , and 113 in the form of bookshelves including a mounting space 114 may be used.

Frame modules

111 , 112 , and 113 may be manufactured by arranging a plurality of beam-shaped sections 115 in a grid to partition the mounting space 114 .

The

frame modules

111 , 112 , and 113 are a first frame module 111 in which the mounting spaces 114 are arranged side by side in the horizontal direction, and a second frame module for mounting a large product 160 such as the display device 161 . (112) and the mounting space 114 in the vertical direction may include a third frame module 113 arranged side by side.

The second frame module 112 includes a large space to be mounted regardless of the size of the display device 161 . If necessary, as shown in FIG. 4 , a beam partitioning the mounting space 114 may be provided to fix the display device 161 and the rigidity of the frame structure 110 may be improved.

The first frame module 111 may be vertically stacked with the second frame module 112 . The horizontal length of the first frame module 111 may be the same as the horizontal length of the second frame module 112 so that the lattice does not shift when stacked in the vertical direction.

The first frame module 111 may be disposed above and below the second frame module 112 , respectively, depending on the size of the first frame module 111 and the second frame module 112 or the size of the installation space. Accordingly, the number of the first frame modules 111 stacked in the vertical direction may be increased or decreased.

The third frame module 113 in which the mounting spaces 114 are arranged side by side in the vertical direction may have the same height as that of the first frame module 111 and the second frame module 112 stacked in the vertical direction. The third frame module 113 may be positioned in the lateral direction of the first frame module 111 and the second frame module 112 as shown in FIG. 4 . A plurality of

frame modules

111 , 112 , and 113 may be combined with fasteners to form a rectangular parallelepiped frame structure 110 .

5 is a view showing a method of manufacturing the

frame modules

111, 112, and 113 of the present invention. The third frame module 113 in which the mounting space 114 is disposed in the vertical direction will be described as an example. Each of the

frame modules

111 , 112 , and 113 may use a section steel 115 elongated in one direction. A shape steel is a structural member and refers to a bar-shaped member having various cross-sectional shapes. A column-shaped section having the same cross section in the longitudinal direction can be constructed by rolling or bending a plate material. Since the section steel has more empty space than a beam-shaped member having a circular or rectangular cross section, it has the advantage of obtaining rigidity of a certain size or more with light and less material.

The present invention may use a C-shaped steel 115 having a C-shaped cross section as shown in FIG. 5 . The C-shaped steel 115 may be formed by vertically bending both ends of a metal plate in the width direction. If both ends are vertically bent only once, a C-shaped cross-section can be obtained, and when both ends are vertically bent twice, a C-shaped cross-section can be obtained. In the inner space, a power source or signal line connected to the electronic product 160 may be disposed, and an outlet module 140 to which power or data cables are fastened may be disposed.

The outlet module 140 may include a power terminal for supplying power, a data terminal for transmitting and receiving data such as an Internet line or cable input, and a connector for connecting a functional unit inserted into the frame to another functional unit. The connector has terminals on both sides of the frame, and data can be exchanged with each other by inserting a data connection line of each electronic device into the terminal. The outlet module 140 may include a rail structure that moves along the

frame modules

111 , 112 , and 113 .

A vent hole 116 may be formed in the section steel 115 as shown in FIG. 5 . The vent hole 116 may reduce the weight of the section steel 115 and may discharge heat generated from the electronic product 160 . In order to more actively dissipate heat, a fan may be used to exhaust hot air to the outside. When the heat of the electronic product 160 is discharged into the room, a heating effect can be obtained in winter. However, in summer, a structure that circulates heat inside using a bar fan that raises the indoor temperature or discharges heat to the outside can be added in addition to wiring connected to the outdoor unit of the air conditioner.

Each frame module (111, 112, 113) may include an outer steel forming a square on the outer periphery and an inner steel for partitioning the mounting space 114 therebetween. The outer cavity may include a vertical cavity extending in a vertical direction and a horizontal cavity extending in a horizontal direction. The first frame module 111 to the third frame module 113 may be assembled by arranging a horizontal steel between a pair of vertical steel. In order to fasten the vertical steel with the fixing member 120 fixed to the ceiling and the floor, it is necessary to arrange the upper and lower ends of the vertical steel to be exposed. Accordingly, the

vertical brackets

122 and 123 of the fixing member 120 may be inserted and fastened to the ends of the vertical steel by locating the vertical steel outside the horizontal steel.

The inner steel located inside the outer steel divides the space surrounded by the outer steel to constitute the mounting space 114, and the inner steel can be welded to the outer steel and assembled to be immovable, and some inner steel is formed using bolts or hooks. It can be combined so that it can be removed as needed.

As described above, if the method of fastening the

frame modules

111 , 112 , 113 in the form of the semi-finished product 160 is used, the number of times of assembly in the field can be minimized. The dimensions of the product 160 can be kept constant and the installation time can be reduced. In addition, it can be modularized to a size that is easy to move, so it can be easily transported to the field.

6 shows an example of various frame structures 110 according to a combination of the

frame modules

111, 112, and 113 of the present invention. 6A is a frame structure 110 in a state in which a pair of first frame modules 111a, one second frame module 112a, and a pair of third frame modules 113 of FIG. 4 are assembled. ) is shown. Since the first frame module 111a has four mounting spaces 114, it may be difficult to install in a narrow indoor space due to a wide width.

Therefore, as shown in (b) of FIG. 6 , a first frame module 111b having a short horizontal length having three mounting spaces 114 may be used. The second frame module 112b may have a length corresponding to the horizontal length of the first frame module 111b.

In addition, the third frame module 113 may be disposed on both sides of the first frame module 111 and the second frame module 112 stacked in the vertical direction. Since the floor height of a typical house is 220 cm to 230 cm, the height of the third frame module 113 may be configured to be less than 220 cm. Although a 5-stage vertical frame is shown in the drawing, a 6-stage vertical frame may also be manufactured in case of a high multi-story or high floor.

When installing in a narrower space, the frame structure 110 may be configured by omitting the third frame module 113 and using only the first frame module 111a and the second frame module 112a, and a large display device ( If the 161) is not mounted, only the third frame may be arranged in a line to implement the frame structure 110 . Each of the

frame modules

111 , 112 , and 113 may be fastened by bolts or by a welding method.

7 is a view for explaining a method of installing the frame structure 110 of the smart wall 100 according to an embodiment of the present invention. The frame structure 110 configured by fastening the

frame modules

111 , 112 , and 113 may be fixed to the ceiling and the floor in order to be installed in an indoor space.

The fixing bracket 125 installed on the ceiling or floor is a member bent in an L-shape, and one side 125b extends vertically from the ceiling or floor, and the other side 125a extends from the ceiling or floor. It may include a first vertical bracket 122 inserted into the inner space of the vertical cavity of the first frame module 111 and the second frame module 112 .

The first vertical bracket 122 may be formed to be longer than the vertical length of the first frame module 111 so that not only the first frame module 111 but also the second frame module 112 can be fastened. A second vertical bracket 123 disposed side by side with the first vertical bracket 122 and inserted into the third frame module 113 may be further included. Since the vertical cavity of the third frame module 113 is one member, it is not necessary to be as long as the first vertical bracket 122 .

The coupling bracket 121 coupled to the ends of the first vertical bracket 122 and the second vertical bracket 123 may form a T-shape together with the

vertical brackets

122 and 123 . The coupling bracket 121 is coupled to the

vertical brackets

122 and 123 and is perpendicular to the seating portion 121b and the seating portion 121b that are seated on the upper surfaces of the first frame module 111 and the third frame module 113 . It may include a fastening portion (121a) extending in the direction and fastened to the fixing bracket (125). The coupling bracket 121 and the fixing bracket 125 can be fastened using a screw, and the fastener to which the screw is fastened can be formed elongated in the vertical direction so that it can be adjusted according to the ceiling height.

When the smart wall 100 is installed on the existing wall, it may further include a fixing member 120 for fastening to the existing wall.

Since the smart wall 100 has an audio device, such as the speaker 162 , when it is used as a temporary wall, a space separated from the wall is generated. At this time, in order to block the sound of the two spaces, the smart wall 100 may be fixed by adding a soundproof structure so that the sound does not leak to the ceiling, floor, and sidewall in contact with the smart wall 100 .

8 is a view for explaining the basket 130 of the smart wall 100 according to an embodiment of the present invention. The

frame modules

111 , 112 , and 113 may be partitioned into a plurality of mounting spaces 114 by using the inner section steel 115 . As shown in FIG. 7 , the basket 130 may be inserted into the mounting space 114 to provide a space for the electronic product 160 to be seated. The second frame module 112 has a relatively large mounting space 114 because the display device 161 is located, but the mounting space 114 of the first frame module 111 and the third frame module 113 is It can be formed in a standardized size. If the mounting space 114 of a standardized size is used, the basket 130 can also be standardized, so there is an advantage in that there is no need to unnecessarily manufacture parts of various shapes.

If the size of the mounting space is all the same, it is convenient for standardization, but a larger space may be required depending on the electronic product 160 to be mounted. For example, since the size of the air conditioner and the air purifier are different, the size of the mounting space 114 may be formed as an integer multiple of the basic size in consideration of the difference. For example, based on the first mounting space 1141 having a horizontal size and b, a second mounting space 1142 having an integer multiple of a or a third mounting space having an integer multiple of b It may include a mounting space 1143 .

The horizontal size of the first mounting space 1141 corresponds to the horizontal size of the third frame module 113 , and the vertical size corresponds to the vertical size of the first frame module 111 . Here, the mounting space 114 means an internal space partitioned by the section steel 115 , so the size of the first frame module 111 and the size of the third frame module 113 are compared with the difference due to the thickness of the section steel 115 . is ignored

Since the first frame module 111 has a short vertical length, it may include only the first mounting space 1141 and the second mounting space 1142 , and the third frame module 113 has a short horizontal length. Therefore, only the first mounting space 1141 and the third mounting space 1143 may be included.

9 is a diagram illustrating a screen change according to a size at which the display device 161 of the smart wall 100 is exposed to the front according to an embodiment of the present invention. As shown in (a) of FIG. 9 , only half of the display device 161 is exposed to provide necessary information to the user, or as shown in FIG.

The smart wall 100 of the present invention may include a screen 200 that covers the front surface of the display device 161 . The screen 200 may use a material that can be wound on a roll, such as a fabric material, and covers the front of the display device 161 so that when the display device 161 is not used, the display device 161 is not exposed to the outside. serves as a cover.

The screen 200 may be wound around the driving roller 210 in order to adjust the area of the first region of the screen 200 positioned in front of the display device 161 . If the number of times that the screen 200 is wound around the driving roller 210 is large, the area of the first region is reduced and the display device 161 is exposed to the front.

A part of the screen 200 (a first area) is located in front of the display device 161 by winding the screen 200 around the driving roller 210 half a turn, and a part of the screen 200 (a second area) is a display device. It can be configured to be located on the back of (161). When the area of the second region positioned on the rear surface of the display device 161 is increased, the area of the first region positioned on the front surface of the display device 161 is reduced. The size of the area exposed on the front surface of the display device 161 may be adjusted.

The control unit 180 adjusts the rotation direction of the driving roller 210 to adjust the area of the screen 200 positioned in front of the display device 161, and when the screen 200 reaches a preset position, the driving roller ( 210) to stop the rotation.

The controller may control the image output to the display device 161 according to the position of the screen 200 . For example, in a line view mode as shown in FIG. 9A or a half view mode as shown in FIG. 9B , an image needs to be output to only a part of the display device 161 . When an image is output to the portion where the screen 200 is located, there is a problem of unnecessary power consumption and providing a cropped image to the user.

When the driving roller is rotated, the screen 200 may be pushed from the surface of the driving roller, and in the case of an embodiment wound on the driving roller, the longer the length of the screen 200 wound on the driving roller, the longer the screen moves per rotation of the driving roller ( Since the length of the 200) is different, it is difficult to accurately match the moving distance of the screen 200 with the number of revolutions of the motor that rotates the driving roller 210 .

In addition, in order to increase the synchronization rate with the position of the screen 200 and the image output to the display device 9161 , a screen sensing unit for detecting the size of the first area of the screen 200 is required. Since the screen 200 of the present invention moves only in the vertical direction, the size of the screen 200 can be known by sensing the movement distance in the vertical direction or the position of the lower end of the first area of the screen 200, the screen sensing unit It can be said that it is a device for sensing the length of the first area in the vertical direction.

10 is a view showing an example of a screen sensing unit of the smart wall 100. The screen sensing unit of this embodiment includes a conductive pattern 241 formed on the surface of the screen 200, a pair of terminals 242, and a pair of and a determination unit that measures the resistance between the terminals 242 and determines that the larger the resistance the larger the size of the first area, and the smaller the resistance the smaller the size of the first area is.

When the pair of conductive patterns 241 are formed on the second surface 200b opposite to the first surface 200a exposed to the outside, they are not exposed to the user. The conductive pattern 241 formed on the second surface 200b includes a first vertical pattern 241a extending in a vertical direction, a second vertical pattern spaced apart from the first vertical pattern 241a and extending in a vertical direction, and the second vertical pattern 241a. A connection pattern 241c connecting the lower end of the first vertical pattern 241a and the lower end of the second vertical pattern 241b may be included.

The positions of the first vertical pattern 241a and the second vertical pattern 241b are not limited on the second surface 200b of the screen 200, and the connection state with the terminal 242 is determined in both the full view state and the line view state. It is sufficient if it has a length that can be maintained, and the connection pattern 241c has a straight line extending in the horizontal direction as shown in the drawing, but it is not necessarily a straight line. The first vertical pattern 241a and the second vertical pattern 241b A conductive member connecting the lower ends of the

The pair of terminals 242 respectively contact the first vertical pattern 241a and the second vertical pattern 241b and are fixed to the frame. To prevent being caught on the terminal 242 and affect the vertical movement of the screen 200, and to maintain the contact state between the conductive pattern 241 and the terminal 242 even if the screen 200 shakes in the front and rear directions For this purpose, a member having elasticity may be used. The contact state between the conductive pattern 241 and the terminal 242 may be maintained by using a material having elasticity such as a sponge or using a member having elasticity such as a clip shape or a pogo pin.

The pair of terminals 242 are connected to the conductive pattern 241 , and when power is applied to the terminal 242 , a current flows through the conductive pattern 241 . As shown in FIG. 10C , the resistance between the terminals 242 may vary according to the length of the conductive pattern 241 . The screen sensing unit may detect the size of the resistance to calculate an increase/decrease in the length of the conductive pattern 241 and calculate the size of the first region of the screen 200 .

11 is a view for explaining the operation of the screen sensor according to a change in the size of the first area of the screen sensor of FIG. 10 . As shown in FIG. 11 (a), the lower end of the display device 161 as shown in FIG. 9 (b) It is a view showing the second surface 200b of the screen 200 in the line view mode in which only a portion is exposed. 11B is a diagram illustrating the second surface 200b of the screen 200 in a half-view mode in which only half of the display unit is exposed as shown in FIG. 9A . FIG. 11C is a diagram illustrating the second surface 200b of the screen 200 in the full view mode exposing the entire area of the display device 161 as shown in FIG. 1 .

When the screen 200 moves in the vertical direction, the contact positions of the pair of terminals 242 on the first vertical pattern 241a and the second vertical pattern 241b may be different. The length of the conductive pattern 241 positioned between the pair of terminals 242 in FIG. 11A is the longest, and the conductive pattern 241 positioned between the pair of terminals 242 in FIG. ) is the shortest. Accordingly, since the length of the conductive pattern 241 positioned between the pair of terminals 242 is changed in each mode, the size of the resistance is also changed. Since the resistance increases as the length of the conductive pattern 241 increases, R1 may be the largest, then R2 and then R3 may decrease in size. The controller may store the resistance value at the designated stop position in the memory, and stop driving the motor when the resistance value stored in the memory is equal to the resistance between the pair of terminals 242 .

12 is a diagram illustrating a modified example of the screen sensing unit of FIG. 10 . Since the screen 200 uses a thin material such as fabric, it may be shaken or wrinkled when the screen 200 moves in the vertical direction. The load bar 209 may be disposed at the lower end of the first region to prevent such wrinkling and maintain a flat state. The load bar 209 may provide a force to pull the first area of the screen 200 downward and prevent the screen 200 from being bent in the horizontal direction. Since the first surface 200a is exposed to the user, it may be disposed on the second surface 200b as shown in FIG. 12 .

The first vertical pattern 241a and the second vertical pattern 241b of the conductive pattern 241 may be disposed adjacent to the left and right ends of the screen 200 as shown in FIG. 12 . An electromagnet 246 may be disposed on a frame adjacent to the first vertical pattern 241a and the second vertical pattern 241b, and a horizontal pulling force may be provided to the screen 200 using the electromagnet 246 .

When the screen 200 moves in the vertical direction, that is, when the motor is driven, the control unit does not apply power to the electromagnet 246 , and the position of the screen 200 can be sensed using the conductive pattern 241 . . When the screen 200 stops moving, that is, when the motor stops, the control unit applies power to the electromagnet 246 , the electromagnet 246 provides a magnetism that pulls the conductive pattern 241 so that the screen 200 is shown in FIG. 12 . As shown in Fig. 1, the attractive force is provided in the left and right directions, and the screen 200 can be spread flat. A pair of electromagnets 246 positioned on the left and right of the screen 200 may apply a force in the left and right directions so that the screen 200 can be flattened regardless of the stop position of the screen 200 .

13 is a diagram illustrating another example of the screen sensing unit of the smart wall 100 . The screen sensor according to the present embodiment may include an ultrasonic sensor 243 for detecting the position of the lower end of the first area. The ultrasonic sensor 243 may measure the distance between the lower end of the screen 200 and the ultrasonic sensor 243 by emitting ultrasonic waves and detecting ultrasonic waves that are reflected and returned to the lower portion of the first region of the screen 200 . Based on this, the size of the first area of the screen 200 may be calculated, and the ultrasonic sensor 243 may be located below the mounting space 114 . In addition to ultrasound, a distance sensing sensor can be implemented using infrared or other microwaves.

14 is a diagram illustrating a specific structure of the screen sensing unit of FIG. 13 . The ultrasonic sensor 243 includes a transmitter 243a that emits ultrasonic waves, and a receiver 243b that detects ultrasonic waves emitted from the transmitter 243a and reflected from the lower end of the first region.

14 (a), the ultrasonic sensor 243 is accommodated in the housing 244 in which the opening 244a is formed in order to prevent foreign substances from entering the ultrasonic sensor 243 when the ultrasonic wave is not used, and The opening 244a can be selectively opened and closed by using the sensor cover 245 . The sensor cover 245 of this embodiment rotates and moves, and the control unit removes the sensor cover 245 only when the motor rotating the driving roller is driven (FIG. 14a), and normally the sensor cover 245 opens the opening. 244a can be maintained in a covered state (FIG. 14(b)).

When the screen sensor is used, the size of the first area of the screen 200 can be continuously monitored, so that the screen 200 can be controlled to be driven naturally. As shown in (a) of FIG. 13, when the first movement starts, the speed of the motor gradually increases from 0 (acceleration section, v1), moves at a constant speed (constant speed section, v2), and reaches a stop position (SP). As it approaches, the speed is reduced (v3, deceleration section) to stop exactly at the stop position SP as shown in FIG. 13(b).

The smart wall 100 of the present invention as described above is mounted on the wall with multimedia devices such as TV and audio, and is installed on the wall at the same height as the wall to provide a neat appearance.

A cleaner appearance of the smart wall 100 may be provided by using the screen 200 selectively exposing the display device 161 according to the usage aspect of the display device 161 .

In addition, since the position of the screen 200 can be accurately detected, the controller can provide an image synchronized with the position of the screen 200 . The above detailed description should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Also, the smart wall and its control method will be described in detail below with reference to FIGS. 15 to 28 . However, those skilled in the art may supplementally interpret FIGS. 15 to 28 or modify the embodiment with reference to FIGS. 1 to 14 described previously.

15 , the smart wall 100 includes a microphone 122 , an air purifier 130 , a first outlet 132 , a second outlet 134 , a first camera 140 , and a second camera 142 . ), display 151 , screen 152 , driving roller 153 , motor 154 , screen sensing unit 155 , first speaker 160 , second speaker 162 , memory 170 , control unit (180).

The smart wall 100 includes a mounting cell for mounting a multimedia device that reproduces content. Here, the multimedia device includes a digital TV, a DVD player, an audio player, and the like.

The microphone 122 processes an external sound signal as electrical voice data. The processed voice data may be utilized in various ways according to a function (or a running application program) being performed by the smart wall 100 . Meanwhile, various noise removal algorithms for removing noise generated in the process of receiving an external sound signal may be implemented in the microphone 122 .

The air purifier 130 purifies the air when the sensed concentration of the fine dust is greater than or equal to a threshold value.

The first outlet 132 discharges the purified air to the first space (discharge air).

The second outlet 134 discharges the purified air to the second space.

The first camera 140 captures at least one first image including a user and an object in front of the screen.

The second camera 142 captures at least one second image including the user and the object.

The first camera 140 and the second camera 142 include a TOF camera.

TOF camera refers to a camera in which a TOF sensor is attached to a general camera. Specifically, an image sensor that captures a scene and shows it produces a 2D-based result. If the TOF sensor intervenes here, depth measurement becomes possible and 3D results can be realized. TOF (Time of Flight) refers to the time it takes for a sound wave or light source to be transmitted to a subject and return to the subject through the subject. The TOF sensor is a kind of component that can detect this role.

A depth map refers to an image containing information related to a distance from an observation point to an object surface in 3D computer graphics.

The TOF camera captures the first image including the depth map. When shooting with a TOF camera, an image is obtained as a result of applying the original image, the depth map, and the depth map to the original image.

The first speaker 160 outputs sound to the first space. The second speaker 162 has a property different from that of the first speaker and includes a second speaker 162 that outputs sound to the second space.

For example, the first speaker 160 may be a 5.1ch non-directional speaker, and the second speaker 162 may be a directional speaker. A topographic speaker is a speaker system that emits sound only in a desired direction, and means that two or more speakers of the same caliber are placed on the same plane to operate as one speaker as a whole. It is also called column speaker because its height is large compared to its width or depth and it is in the shape of a column. It does not occupy a floor area and has the advantage of being able to adjust the listening position to the height of the ear.

The memory 170 stores a user's preset motion image.

The controller 180 determines whether to switch to the directional speaker based on the images of the first camera 140 and the second camera 142 , and controls at least one of the first outlet 130 and the second outlet 132 . do.

The display 151 displays a graphic image. The controller 180 controls the display 151 to display a graphic image of content reproduced by the multimedia device mounted on the smart wall 100 .

The screen 152 covers the front surface of the display and is rolled over the display 151 to adjust the screen size of the display 151 . 19 , the screen size of the display may be a line view, a half view, or a full view.

The driving roller 153 is located above the screen and controls the screen to be wound.

The motor 154 provides rotational force to the driving roller 153 .

The screen sensing unit 155 detects the size of the first area of the screen.

The memory 170 stores an image of an object.

The controller 180 controls the driving of the motor based on the size of the first region sensed by the screen sensing unit 155, and based on the at least one first image and the at least one second image, Recognizing a motion, recognizing the object based on at least one first image and the at least one second image, determining a user's intention based on the recognized motion of the user and the recognized object, and determined The multimedia device is controlled based on the user's intention.

16 is a flowchart illustrating a method for controlling a smart wall according to an embodiment of the present invention. The present invention is performed by the controller 180 .

At least one first image including the user and the object in front is captured by the first camera (S1610).

At least one second image including the user and the object is captured by a second camera (S1620).

A user's motion is recognized based on the at least one first image and the at least one second image (S1630).

The object is recognized based on the at least one first image and the at least one second image (S1640).

The user's intention is determined based on the recognized user's motion and the recognized object (S1650).

The multimedia device is controlled based on the determined user's intention (S1660).

Referring to FIG. 17 , the smart wall 100 includes a screen 152 and a display 151 . As the screen 152 moves up and down on the display 151 , the size of the area viewed by the user may be adjusted by the display 151 .

Referring to FIG. 18 , a mounting space of the smart wall 100 will be described. The mounting space of the smart wall 100 means a space for mounting parts.

The mounting space of the smart wall 100 includes a driving roller 153 , a first speaker 160 , a second speaker 162 , a display 151 , a first camera 140 , a second camera 142 , and an air purifier. A first outlet 132 connected to the second outlet 134 connected to the air purifier may be equipped with a device cabinet (not shown), a power distributor (not shown), an internet distributor (not shown), and the like.

19 is a diagram illustrating a line view, a half view, and a full view of a smart wall according to an embodiment of the present invention. 19 includes FIGS. 19(a), 19(b), and 19(c).

19A is a diagram illustrating a line view. 19A is a diagram illustrating a line view. 19B is a diagram illustrating a half view. 19( c ) is a diagram illustrating a full view.

Referring to FIG. 19A , the line view refers to a view in which the display 151 in the smallest size is exposed at the bottom of the smart wall and provides simple notification information.

Referring to FIG. 19(b) , the half view refers to a view in which the display 151 is exposed in a medium size when the smart wall is stopped, and provides notification information with a larger amount of information.

Referring to FIG. 19(c) , the full view refers to a view used when the display 151 is exposed in the largest size when the smart wall is stopped, and is used when watching a movie or watching TV.

20 is a diagram illustrating user recognition, user motion recognition, and object recognition with a camera mounted on a smart wall, according to an embodiment of the present invention. Fig. 20 includes Figs. 20(a) and 20(b).

FIG. 20A is a diagram illustrating determination of a user's intention through user recognition, user gesture recognition, and object recognition. FIG. 20(b) is a diagram illustrating pause of content reproduced by a multimedia device.

Referring to FIG. 20 ( a ), when the user 10 holds the teacup object 25 and sits on the sofa object 21 , the first camera 140 captures the first image, and the second camera 142 captures a second image.

The controller 180 recognizes a user's motion based on the first image and the second image, and recognizes an object.

The controller 180 recognizes the user based on the first image and the second image, recognizes the user's motion, and recognizes an object.

When there are a plurality of users, an embodiment of recognizing users individually will be described.

The controller 180 may recognize a user based on at least one first image and at least one second image, and individually determine the user's intention based on the recognized user. Here, the first image and the second image may be at least one first image and at least one second image.

For example, the controller 180 recognizes that the user 10 is the mother 2010 among family members based on the first image and the second image. Family members include father, mother and baby.

The controller 180 recognizes that the user's action 2020 is to sit on the sofa 21 based on the first image and the second image. The controller 180 recognizes the object 2030 based on the first image and the second image. For example, the recognized object 2030 may be the sofa 21 and the teacup 25 .

The controller 180 determines the user's intention based on the recognized user 2010 , the recognized user's action 2020 , and the recognized object 2030 .

Referring to FIG. 20( b ), the object 20 includes a sofa 21 , a table 22 , a smartphone 23 , a book 24 , and a teacup set 25 .

In the case of object 20 recognition, in order to increase the recognition rate, the image of the object 20 is stored in the memory 180 , and the controller 180 refers to the image of the object 20 stored in the memory 170 , in FIG. As shown in 21(a), the object 2030 is recognized by comparing the first image and the second image.

An embodiment of determining a user's intention will be described with reference to FIG. 20A .

For example, the user 2010 is a wife among family members, the user's action 2020 is the action of sitting on the sofa holding a teacup in his hand, and the recognized object 2030 is the teacup 25 and the sofa 21 . In this case, the controller 180 determines that the user's intention is that the wife sits on the sofa and rests.

For example, if the user 2010 is a husband among family members, the user's motion 2020 is the motion of sitting on the sofa, leaning back and making an arm pillow with both arms, and the recognized object 2030 is the sofa 21 , the controller 180 determines that the user's intention is to watch the TV mounted on the smart wall 100 while the husband sits on the sofa.

For example, if the user 2010 is a baby among family members, the user's motion 2020 is a sitting motion on the floor, and the recognized object 2030 is a toy (not shown), the controller 180 controls the user's intention decides that the baby is sitting on the floor and playing with the toy.

According to an embodiment of the present invention, in the case of object recognition, in order to increase the recognition rate, an object recognition target is designated in advance, and the external characteristics of the object are learned in advance using an AI function.

Next, an embodiment (not shown) of executing the confirmation procedure by voice in order to increase the accuracy of the determination of the user's intention will be described.

With respect to determining the user's intent, the initial accuracy may be low. Therefore, in order to increase the accuracy of the determination of the user's intention, the confirmation procedure may be initially performed by voice.

The microphone 122 receives the user's voice from the user. The first speaker 160 outputs a sound corresponding to the content according to a control command from the controller 180 .

The controller 180 controls the first speaker 160 to output a first voice message for confirming the user's intention, and when a feedback corresponding to the first voice message is received from the user 10, based on the received feedback to control the multimedia device.

A method of confirming whether the determined intention of the user is correct will be described. In the controller 180, will the first speaker 160 play music? In the case of outputting a voice message such as , when the user utters 'Play music', the user's feedback is received through a microphone, and the multimedia device is controlled to play music based on the received user's feedback.

When this pattern is repeated a predetermined number of times, the controller 180 determines the user's intention to sit on the sofa and take a break, and automatically controls the multimedia device to play music.

For example, if the user 2010 is a husband among family members, the user's motion 2020 is the motion of sitting on the sofa, leaning back and making an arm pillow with both arms, and the recognized object 2030 is the sofa 21 , the controller 180 determines that the user's intention is that the husband sits on the sofa and watches TV with the multimedia device mounted on the smart wall 100 .

A method of confirming whether the determined intention of the user is correct will be described. Does the controller 180 turn on the TV through the first speaker 160? In the case of outputting a voice message such as , when the user utters 'Turn on the TV', the user's feedback is received through the microphone, and the multimedia device controls the TV to be played based on the received user's feedback.

When this pattern is repeated a predetermined number of times, the controller 180 determines the user's intention to watch TV with the multimedia device mounted on the smart wall 100 while sitting on the sofa, and automatically causes the multimedia device to play the TV program. Control.

21 is a diagram illustrating setting of an environment of a multimedia device based on a user's intention according to an embodiment of the present invention. Fig. 21 includes Figs. 21(a) and 21(b).

21(a) is a diagram illustrating determination of a user's intention. 21( b ) is a diagram illustrating setting of an environment of a multimedia device based on a determined intention of a user.

Referring to FIG. 21 ( a ), the user 2110 is a wife among family members, and the user's operation 2120 is an operation to sit on the sofa 21 holding the teacup 25 in her hand and read the book 26 , If the recognized object is the teacup 25 , the sofa 21 , and the book 26 , the controller 180 determines that the user's intention is that the wife sits on the sofa and reads the book.

Referring to FIG. 21B , the controller 180 sets the environment of the multimedia device based on the determined intention of the user.

For example, if it is determined that the wife 2110 sits on the sofa 21 and reads the book 26 while holding the teacup 25 in her hand, the controller 180 sets the environment of the multimedia device to play soft music. and the display 151 presents the audio player to be displayed in line view.

According to the present invention, it is possible to automatically set the multimedia device along with natural motions based on the learned usage patterns for each family member.

22 is a diagram illustrating setting of an environment of a multimedia device based on a user's intention according to an embodiment of the present invention. Fig. 22 includes Figs. 22(a) and 22(b).

22( a ) is a diagram illustrating determination of a user's intention. 22( b ) is a diagram illustrating setting of an environment of a multimedia device based on a determined intention of a user.

Referring to FIG. 22( a ), the user 2210 is a husband among family members, the user's operation 2220 is an operation of sitting on the sofa, leaning back and making an arm pillow with both arms, and the recognized object is the sofa 21 ), the controller 180 determines that the user's intention is that the husband sits on the sofa 21 and watches TV with the multimedia device mounted on the smart wall 100 .

Referring to FIG. 22B , the controller 180 sets the environment of the multimedia device based on the determined intention of the user.

For example, when it is determined that the husband 2210 sits on the sofa 21, leans back, and puts a foot pillow with both arms, the controller 180 sets the environment of the multimedia device to watch TV, and the display 151 ) controls to display the TV screen in full view.

23 is a diagram illustrating setting of an environment of a multimedia device based on a user's intention according to an embodiment of the present invention. Fig. 23 includes Figs. 23(a) and 23(b).

23(a) is a diagram illustrating determination of a user's intention. 23( b ) is a diagram illustrating setting of an environment of a multimedia device based on a determined intention of a user.

Referring to FIG. 23(a) , the user 2310 is a baby among family members, the user's operation 2220 is an operation of sitting on the sofa, leaning back and using both arms to make an arm pillow, and the recognized object is the sofa 21 ), the controller 180 determines that the user's intention is that the husband sits on the sofa 21 and watches TV with the multimedia device mounted on the smart wall 100 .

For example, the user 2310 is a baby among family members, the user's action 2320 is the action of sitting in front of the sofa 21 and playing with the toy 27, and the recognized object is the sofa 21, the toy ( 27), the controller 180 determines that the user's intention is that the baby 2310 sits in front of the sofa 21 and plays with the toy 27 .

Referring to FIG. 23B , the controller 180 sets the environment of the multimedia device based on the determined intention of the user.

For example, when it is determined that the baby 2310 sits in front of the sofa 21 and plays with the toy 27, the controller 180 sets the environment of the multimedia device to watch a baby program in which the baby plays with them, , the display 151 controls to display the baby program to play with in a half view.

If there are a plurality of users, it may be a problem as to who should set the multimedia device environment.

For example, there is at least two of a husband, a wife, and a baby in the first image and the second image. Determined based on weight. For example, 10 babies, 5 wives, and 1 husband. Weights can be adjusted by user settings.

When a husband, a wife, and a baby are present in the first image and the second image at the same time, the controller 180 may set the multimedia device environment based on the baby because the weight of the baby is the highest.

24 is a diagram illustrating adjusting a volume of a speaker when a user makes a phone call, according to an embodiment of the present invention. Fig. 24 includes Figs. 24(a) and 24(b).

24A is a diagram illustrating a user making a phone call. 24 (b) is a diagram illustrating adjusting the volume of the speaker.

The first speaker 160 and the second speaker 162 output sound output when the multimedia device mounted on the smart wall 100 reproduces content.

When the user 10 performs a specific action, the controller 180 controls the volume of the sound output from the first speaker 160 to be adjusted.

For example, when the user makes a phone call, the controller 180 reduces the volume 2410 of the sound output from the first speaker 160 . Alternatively, the mute function can be executed.

The controller 180 may determine based on the first image and the second image that the user's intention is that the user 10 wants to make a phone call while holding the smartphone 23 on the sofa 21 .

When it is determined based on the first image and the second image that the user has ended the phone call, the controller 180 may control the volume of the sound output from the first speaker 160 to return to its original size.

25 is a diagram illustrating pause of content reproduced by a multimedia device when a user disappears from a camera angle of view, according to an embodiment of the present invention. Fig. 25 includes Figs. 25(a) and 25(b).

Figure 25 (a) is a diagram illustrating that the user gets up from the sofa and disappears. 25(b) is a diagram illustrating pause of content reproduced by a multimedia device.

When the user 10 disappears from the angle of view of the camera while the multimedia device mounted on the smart wall 100 reproduces content, the controller 180 controls to pause (2510) the content reproduced by the multimedia device.

For example, while the user 10 sits on the sofa 21 and watches content reproduced by the multimedia device mounted on the smart wall 100, if the user 10 gets up from the sofa 21 for a while and moves to another place, the first image and the second image do not include an image of the user 10 .

The controller 180 may determine based on the first image and the second image that the user's intention wants the user 10 to get up from the sofa 21 and do something more important than watching the content. When the user 10 disappears from the angle of view of the camera while the multimedia device is playing the content, the controller 180 controls the content played by the multimedia device to pause ( 2510 ).

In addition, when the user returns to the original position, the automatic content playback will be described. When it is determined that the user 10 returns and sits on the sofa 21 based on the first image and the second image, the controller 180 may control the content reproduced by the multimedia device to be replayed from the point at which it is paused. .

Next, with reference to FIG. 25 , when the user sleeps, an embodiment in which a screen for reproducing content is paused, a predetermined time elapses, and the power is turned off will be described.

When the user 10 sleeps while the multimedia device is playing content, the controller 180 controls the content 2510 played by the multimedia device to pause. When it is determined that the user 10 continues to sleep after a predetermined time has elapsed from the moment of pause, the multimedia device is controlled to be turned off.

When the user 10 does not move for a predetermined time or more based on the first image and the second image, the controller 180 determines that the user 10 sleeps.

Referring to Figure 26, the reinforcement learning model (300, Reinforcement Learning, hereinafter, RL) is one of the problems that machine learning deals with, and the agent 301 defined in an environment 302 recognizes the current state 303, , means a method of selecting the action that maximizes the reward 305 among the selectable actions 304 .

The reinforcement learning model 300 includes an agent 301 , an environment 302 , a state 303 , a reward 304 , and an action 305 .

Agent 301 (Agent) means observing state 303, selecting action 305, and directing a goal. The agent 301 includes a learner and a decision maker.

The environment 302 (Environment) means the rest except for the agent 301 .

A state 303 (State) means information indicating the current situation of a system user.

Action 304, Action means what the agent 301 does in the current situation.

The reward 305 (Reward) refers to information indicating good or bad behavior 304 .

An embodiment of determining a user's intention using a reinforcement learning model reflecting feedback will be described with reference to FIG. 26 .

The controller 180 receives feedback from the user and determines the user's intention by using the reinforcement learning model 300 reflecting the received feedback. Feedback includes positive feedback and negative feedback. Here, the feedback corresponds to the reward 305 .

A case where the user is a wife will be described.

For example, as shown in FIG. 21 , the user 2110 is a wife among family members, and the user's action 2120 is to sit on the sofa 21 with the teacup 25 in his hand and read the book 26 . If it is an action, and the recognized object is a teacup 25 , a sofa 21 , and a book 26 , the controller 180 determines that the user's intention is that the wife sits on the sofa and reads the book.

The controller 180 sets the environment of the multimedia device to reproduce calm music based on the user's intention, and presents the display 151 to display the audio player in a line view.

If the user 10 does not change the settings of the multimedia device, the controller 180 determines that the user's intention is correct. When the user 10 changes the settings of the multimedia device, the controller 180 determines that the user's intention is wrong.

If the determined intention of the user is correct, the controller 180 provides additional points to the reinforcement learning model 300 , and if the determined intention of the user is incorrect, the controller 180 provides points to the reinforcement learning model 300 .

For example, if the determined intention of the user is correct, the controller 180 provides an additional point + 10 to the reinforcement learning model 300 , and if the determined intention of the user is incorrect, the controller 180 provides a deduction point - 10 to the reinforcement learning model 300 . to provide.

A case in which the user is a husband will be described.

For example, as shown in FIG. 22 , the user 2210 is a husband among family members, and the user's operation 2220 is an operation of sitting on a sofa, leaning back and placing an arm pillow with both arms, and the recognized object is If it is the sofa 21 , the controller 180 determines that the user's intention is that the husband sits on the sofa 21 and watches TV with the multimedia device mounted on the smart wall 100 .

The controller 180 sets the environment of the multimedia device to watch TV based on the user's intention, and controls the display 151 to display the TV screen in full view.

If the user does not change the settings of the multimedia device, the controller 180 determines that the user's intention is correct. When the user changes the settings of the multimedia device, the controller 180 determines that the user's intention is wrong.

A case in which the user is a baby will be described.

For example, as shown in FIG. 23 , the user 2310 is a baby among family members, the user's action 2320 is the action of sitting in front of the sofa 21 and playing with the toy 27 , and the recognized object If is the sofa 21 and the toy 27 , the controller 180 determines that the user's intention is that the baby 2310 sits in front of the sofa 21 and plays with the toy 27 .

The controller 180 sets the environment of the multimedia device to watch a baby program with which the baby plays with the user based on the user's intention, and controls the display 151 to display the baby program with which the baby plays with the half view.

27 is a diagram illustrating prediction of user behavior based on a 2D image according to an embodiment of the present invention. Fig. 27 includes Figs. 27(a), 27(b) and 27(c).

27( a ) is a diagram illustrating a first action of a user predicted 1 second before and an actual first action of the user. FIG. 27(b) is a diagram illustrating a second action of a user expected 1 second before and an actual second action of the user. FIG. 27( c ) is a diagram illustrating the user's third action predicted 1 second before and the actual user's third action.

Referring to FIG. 27( a ), the screen image is analyzed based on the TV drama screen image, and the action predicted by the controller 180 1 second before is kiss , and the next screen is actually kiss .

Referring to FIG. 27(b) , the screen image is analyzed based on the TV drama screen image, and the action predicted by the controller 180 1 second before is high five, and the next screen is actually high five.

Referring to FIG. 27(c) , the screen image is analyzed based on the TV drama screen image, and the action predicted by the controller 180 one second before is Hug, and the next screen is actually Hug.

According to the present invention, it is possible to train an artificial intelligence using a TV drama screen and predict what kind of motion or object will appear on the next screen.

An embodiment of recognizing a user's motion based on a current image and a previous image will be described with reference to FIG. 27 .

The controller 180 recognizes the user's motion based on the current image and the previous image maintaining a predetermined time interval among at least one first image.

For example, referring to FIG. 21 , the image 3 seconds before is an image of the user 2110 holding the teacup 25 in his hand, sitting on the sofa 21 and viewing the book 26 . The current image is an image in which the user 2110 continues to view the book 26 while sitting on the sofa 21 with a teacup in his hand.

Accordingly, the controller 180 recognizes that the user 2110 reads the book 26 while sitting on the sofa 21 holding the teacup in his hand based on the current image and the previous image.

For example, referring to FIG. 22 , the image 3 seconds before is an image in which the user 2210 sits on the sofa 21, leans back, and puts an arm pillow with both arms. The current image is an image in which the user 2210 sits on the sofa 21, leans back, and continues the arm pillow with both arms.

Accordingly, the controller 180 recognizes that the user 2210 sits on the sofa 21, leans back, and puts an arm pillow with both arms, based on the current image and the previous image.

For example, referring to FIG. 23 , the image 3 seconds before is an image in which the user 2310 sits in front of the sofa 21 and plays with the toy 27 . The current image is an image in which the user 2310 sits in front of the sofa 21 and continues to play with the toy 27 .

Accordingly, the controller 180 recognizes that the user 2310 sits in front of the sofa 21 and continues playing with the toy 27 based on the current image and the previous image.

Referring to FIG. 28 , when the user 10 sits on the specific object 20 , the camera 140 captures the user's body point and recognizes the user's motion based on the photographed body point. Here, the specific object 20 may be a sofa.

Specifically, when the user 10 moves while holding the object 20 , the controller 180 allows the first camera 140 and the second camera 144 to capture the user's body points 11 , 12 , and 13 . Control and recognize the user's motion based on the captured body points (11, 12, 13).

For example, the body points include a shoulder point 11 , an elbow point 12 , and a neck point 13 .

The controller 180 recognizes the user's motion based on the degree of movement and bending of body points such as the shoulder point 11 and the elbow point 12 . Referring to FIG. 28 , the controller 180 determines that the user 10 sits on the sofa object 20 on the basis of the body point, puts both hands behind the head, and tilts the body back.

The smart wall and its control method according to the present invention are not limited to the configuration and method of the described embodiments as described above, but all or part of each embodiment may be modified so that various modifications can be made. They may be selectively combined and configured.

On the other hand, the smart wall control method of the present invention can be implemented as a processor-readable code on a processor-readable recording medium provided in the smart wall. The processor-readable recording medium includes all types of recording devices in which data readable by the processor is stored. Examples of the processor-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also includes those implemented in the form of carrier waves such as transmission over the Internet . In addition, the processor-readable recording medium is distributed in a computer system connected through a network, so that the processor-readable code can be stored and executed in a distributed manner.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

Various embodiments have been described in the best mode for carrying out the present invention.

The present invention is used in a smart wall related field in which a display mounted on a smart wall provides information related to parts by automatically detecting the state of parts mounted on the smart wall or the presence or absence of abnormalities.

It will be apparent to those skilled in the art that various changes and modifications can be made in the present invention without departing from the spirit or scope of the present invention. Accordingly, it is intended that this invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.

Claims

In a smart wall including a mounting space (mounting cell) for mounting a multimedia device that reproduces content,

a display for displaying a graphic image corresponding to the content;

a screen that covers the front surface of the display and is rolled on the display to adjust a screen size of the display;

a driving roller positioned on the screen and wound around the screen;

a motor providing a rotational force to the driving roller;

a screen sensing unit for sensing the size of the first area of the screen;

a first camera for capturing at least one first image including a user and an object in front of the screen;

a second camera configured to capture at least one second image including the user and the object;

Controlling the driving of the motor based on the size of the first area sensed by the screen sensing unit,

Recognizing a user's motion based on the at least one first image and the at least one second image,

Recognizing the object based on the at least one first image and the at least one second image,

Determining a user's intention based on the recognized user's motion and the recognized object,

Comprising a control unit for controlling the multimedia device based on the determined intention of the user,

smart wall.
According to claim 1, wherein the control unit

Store the image of the object in advance in memory,

Recognizing the object by referring to the image of the object stored in the memory,

smart wall.
The method of claim 1,

a microphone for receiving a user's voice from the user;

Further comprising a speaker for outputting sound,

the control unit

Control the speaker to output a first voice message confirming the user's intention,

When a feedback corresponding to the first voice message is received from the user with the microphone, controlling the multimedia device based on the received feedback,

smart wall.
According to claim 1, wherein the control unit

Recognizing a user based on the at least one first image and the at least one second image,

Individually determining the user's intention based on the recognized user,

smart wall.
5. The method of claim 4, wherein the control unit

Setting the environment of the multimedia device based on the determined intention of the user,

smart wall.
The method of claim 1,

Further comprising a speaker for outputting sound,

the control unit

When the user performs a specific action, controlling to adjust the volume of the sound output by the speaker,

smart wall.
According to claim 1, wherein the control unit

When the user disappears from the view angle of the camera while the multimedia device is playing the content, controlling the multimedia device to pause the content reproduced,

smart wall.
According to claim 1, wherein the control unit

receive feedback from users,

Determining the user's intention using a reinforcement learning model reflecting the received feedback,

smart wall.
9. The method of claim 8,

If the determined intention of the user to be included is correct, a reward point is provided to the reinforcement learning model,

If the determined intention of the user to include is wrong, providing a deduction point to the reinforcement learning model,

smart wall.
According to claim 1, wherein the control unit

Recognizing a user's motion based on a current image and a previous image maintaining a predetermined time interval among the at least one first image,

smart wall.
A method for controlling a smart wall including a mounting cell for mounting a multimedia device that reproduces content, the method comprising:

capturing at least one first image including a user and an object in front of the screen with a first camera;

capturing with a second camera that captures at least one second image including the user and the object;

recognizing a user's motion based on the at least one first image and the at least one second image;

recognizing the object based on the at least one first image and the at least one second image;

determining a user's intention based on the recognized user's motion and the recognized object; and

controlling the multimedia device based on the determined intention of the user

A smart wall control method comprising a.
12. The method of claim 11,

storing the image of the object in advance in a memory; and

By referring to the image of the object stored in the memory, further comprising the step of recognizing the object,

Smart Wall Control Method.
12. The method of claim 11,

controlling the speaker to output a first voice message confirming the user's intention; and

When receiving the feedback corresponding to the first voice message from the user with a microphone, further comprising the step of controlling the multimedia device based on the received feedback,

Smart Wall Control Method.
12. The method of claim 11,

recognizing a user based on the at least one first image and the at least one second image; and

Further comprising the step of individually determining the user's intention based on the recognized user,

Smart Wall Control Method.
15. The method of claim 14,

Further comprising the step of setting the environment of the multimedia device based on the determined intention of the user,

Smart Wall Control Method.
12. The method of claim 11,

When the user performs a specific action, further comprising the step of controlling to adjust the volume of the sound output by the speaker,

Smart Wall Control Method.
12. The method of claim 11,

When the user disappears from the angle of view of the camera while the multimedia device is playing the content, further comprising the step of controlling the multimedia device to pause the content reproduced,

Smart Wall Control Method.
12. The method of claim 11,

receiving feedback from a user; and

Further comprising the step of determining the intention of the user by using a reinforcement learning model reflecting the received feedback,

Smart Wall Control Method.
19. The method of claim 18,

providing reward points to the reinforcement learning model if the determined intention of the user to include is correct; and

If the determined intention of the user to be included is wrong, further comprising the step of providing a deduction point to the reinforcement learning model,

Smart Wall Control Method.
12. The method of claim 11,

The method further comprising the step of recognizing a user's motion based on a current image and a previous image maintaining a predetermined time interval among the at least one first image,

Smart Wall Control Method.