KR20200039995A - Method of space touch detecting and display device performing the same - Google Patents

Abstract

The present invention relates to a display device capable of a three-dimensional touch input. The display device comprises a display panel. A first camera device is disposed in front of the display panel and generates a first side image of a sensing space from a display surface of the display panel to a predetermined distance. A second camera device is disposed to be spaced apart from the first camera device in front of the display panel, and generates a second side image of the sensing space. A control part recognizes a non-contact space touch input in the sensing space based on the first and second side images. The control part comprises an extraction part detecting a first input object placed in the sensing space from the first and second side images, and a calculation part that calculates 3D position information of the first input object based on first position information in the first side image of the first input object and second position information in the second side image.

Description

Spatial touch detection method and display device for performing the same {METHOD OF SPACE TOUCH DETECTING AND DISPLAY DEVICE PERFORMING THE SAME}

The present invention relates to a display device, and more particularly, to a spatial touch sensing method for sensing a spatial touch and a display device performing the same.

A touch screen that is installed on a display device and inputs data by simple contact using a finger or a pen is widely used.

The touch screen detects the occurrence of user input by attaching a touch panel to the display panel and recognizing that the characteristics of the corresponding area (that is, the area in which the finger touches) change when a finger or a pen touches.

Since the touch screen as described above generally adopts a method of operating in a two-dimensional plane in which a display screen and a touch panel are arranged on the same surface, there is a limit in providing various expressions of the user interface.

Accordingly, a problem to be solved by the present invention is to provide a display device capable of detecting a 3D touch input.

Another problem to be solved by the present invention is to provide a spatial touch sensing method capable of detecting a 3D touch input.

The problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A display device according to an exemplary embodiment of the present invention for solving the above problems includes a display panel; A first camera device disposed in front of the display panel and generating a first side image of a sensing space from a display surface of the display panel to a predetermined distance; A second camera device disposed in front of the display panel and spaced apart from the first camera device to generate a second side image of the sensing space; And a controller that recognizes a non-contact spatial touch input in the sensing space based on the first and second side images. Here, the control unit, the extraction unit for detecting a first input object located in the detection space from the first and second side images; And a calculator configured to calculate 3D position information of the first input object based on first position information in the first side image of the first input object and second position information in the second side image.

The extraction unit detects a first object image corresponding to the first input object from the first side image based on a preset pattern, and removes the first object image based on a feature point predefined in the preset pattern. 1 A reference point is determined, and a second object image corresponding to the first input object is detected from the second side image based on a preset pattern, and the second object image is based on a feature point predefined in the preset pattern. Can determine a second reference point.

The calculating unit calculates a first position angle based on the position of the first reference point in the first side image, and calculates a second position angle based on the position of the second reference point in the second side image, A second generator that generates two-dimensional position information of the first input object based on the first position information of the first camera device, the second position information of the second camera device, the first position angle and the second position angle 1 position calculator; And a third position angle based on the position of the first reference point in the first side image, and a fourth position angle based on the position of the second reference point in the second side image, and the first A second position generating depth information of the first input object based on the first position information of the camera device, the second position information of the second camera device, the third position angle and the fourth position angle. It includes a calculation unit, the first position angle relative to the first camera device, an angle between one surface of the sensing space and the first input object, the second position angle relative to the second camera device , An angle between the one surface of the sensing space and the first input object, and the third position angle is an angle between the display panel and the first input object based on the first camera device, and the third The position angle is the first An angle between the display panel and the first input object based on a camera device, and the 3D position information of the first input object may include the 2D position information and the depth information.

The extracting unit includes a tracking unit that tracks a change over time of the first reference point; And a correction unit that corrects the change of the first reference point using a Kalman filter or a low frequency filter.

The control unit may include a background removing unit generating a first pre-processed image by removing a background image from the first side image; A noise removing unit that binarizes the first pre-processed image to generate a first original image, respectively; And selecting one of the candidate object images as a shadow image based on at least one of the size of the candidate object images included in the first original image, the symmetry between the candidate object images, and the placement positions of the candidate object images. , A shadow removing unit generating a first shadow removal image by removing the shadow image from the first original image, and the first shadow removal image may be provided to the extraction unit as the first side image.

When the first input object includes a plurality of sub-objects separated from each other, the extraction unit may select a first sub-object adjacent to one side of the display panel among the sub-objects as a first input object.

The display device may include a third camera device that is disposed away from the first and second camera devices in front of the display panel and generates a third side image of the sensing space; And a fourth camera device, which is spaced apart from the first to third camera devices in front of the display panel, and generates a fourth side image of the sensing space. In this case, the extraction unit detects a second input object located in the sensing space from the third and fourth side images, and the calculation unit includes third location information in the third side image of the second input object. 3D position information of the second input object may be calculated based on the fourth position information in the fourth side image.

The extracting unit may perform shape analysis on at least one of the first side image and the second side image when the partial image of the first input object is short, and the first side image and the When at least one of the second side images is a far-field full image of the first input object, a ratio analysis unit performing ratio analysis on the first input object may be further included.

The spatial touch sensing method according to an exemplary embodiment for solving the above other problems is performed in a display device including first and second camera devices that are spaced apart from each other in front of the display panel. The method includes generating first and second side images of a sensing space from a display surface of the display panel to a predetermined distance through the first and second camera devices; Detecting an input object located in the sensing space from the first and second side images; And calculating 3D position information of the input object based on first position information in the first side image of the input object and second position information in the second side image.

Specific details of other embodiments are included in the detailed description and drawings.

The display device according to the embodiments of the present invention acquires side images of the sensing space in front of the display unit and calculates a three-dimensional position of the input object in the sensing space from the side images, thereby forming a short distance from the display unit. In the 3D touch input can be detected.

The effects according to the embodiments of the present invention are not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a perspective view illustrating a display device according to an exemplary embodiment of the present invention.
2A and 2B are exploded perspective views illustrating an example of a display unit included in the display device of FIG. 1.
2C is a bottom side view illustrating an example of the display device of FIG. 1.
FIG. 2D is an enlarged view of the area Q1 in FIG. 1.
3A to 3C are diagrams illustrating an example of a sensing space formed by an input sensing unit included in the display device of FIG. 1.
4 is a diagram illustrating a display device according to some example embodiments of the present invention.
5 is a plan view illustrating an example of a sensing space formed by the display device of FIG. 4.
6 is a block diagram illustrating an example of a control unit included in the display device of FIG. 1.
7A to 7F are diagrams illustrating a process of detecting an input object in the control unit of FIG. 6.
8A and 8B are schematic views showing an input image of a pointer according to a position.
9 is a schematic diagram showing a comparison of an original image and a converted image of a proximity pointer.
10 is a schematic diagram showing a method of analyzing a ratio of a converted image.
11A and 11B are views illustrating an example of a sensory providing unit included in the display device of FIG. 1.
12 is a view showing a display device according to another exemplary embodiment of the present invention.
13 is a diagram illustrating a configuration in which the display device of FIG. 12 detects multi-input.
14 is a flowchart illustrating an example of a spatial touch sensing method performed in the display device of FIG. 1.
15A is a flowchart illustrating an example of a process in which the method of FIG. 14 preprocesses a side image.
15B is a flowchart illustrating an example of a process in which the method of FIG. 14 detects an input object.
16 is a flowchart illustrating another example of a spatial touch sensing method performed on the display device of FIG. 12.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and the ordinary knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims.

Although the first, second, etc. are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical spirit of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a perspective view illustrating a display device according to an exemplary embodiment.

Referring to FIG. 1, the display device 1 includes a display unit 100, an input detection unit 300, and a control unit 500.

The display 100 may provide an image. Here, one surface of the display unit 100 on which the image is displayed is defined as a display surface, and the upper, lower, left, and right sides of the display device 1 are based on the display surface of the display unit 100 shown in FIG. 1 ( For example, it is defined as an upper side, a lower side, a left side, and a right side, based on the surfaces formed by the first direction D1 and the second direction D2. In addition, the front (or the top) of the display device 1 is defined as a direction in which the display surface is visible based on a direction perpendicular to the display surface (ie, the third direction D3), and the rear of the display device 1 (Or, lower part) is defined in a direction in which the display surface is not visible.

The display unit 100 may have a square planar shape. The display unit 100 may be a curved curved display module having a specific radius of curvature. For example, the display surface of the display unit 100 may be bent with a radius of curvature of about 1.5 m. As shown in FIG. 1, the display unit 100 may have a concave shape from front to rear, or conversely. However, the present invention is not limited thereto, and the display unit 100 may be a flat display module having a flat display surface. Hereinafter, the display unit 100 will be described as an example in the case of a curved display module.

The display unit 100 may be a liquid crystal display (LCD), an organic light emitting display (OLED), a plasma display (PDP), a field emission display (FED), or the like, according to a method of emitting light. A more specific configuration of the display unit 100 will be described later with reference to FIG. 2A.

The input sensing unit 300 may generate sensing data for a sensing space (or sensing area, see FIG. 3A) set in front of the display unit 100. Here, the sensing space has an area equal to or similar to the display surface of the display unit 100 (based on a plane or a curved surface parallel to the display surface) within a predetermined distance (eg, up to 1 m) from the display surface of the display unit 100 As a space to be located, it is a three-dimensional space having a predetermined volume, and may be a space in which a non-contact space touch is input by an input object OBJ. The input object OBJ may include a pointer. The pointer is a pointing means indicating the direction or position, and a user's finger, pen, indicator bar, baton, antenna, stick, etc. can be used as the pointer. In the drawing, as an example of a pointer, a user's hand (or finger) and a pen are illustrated, but it is obvious that examples of applicable pointers are not limited thereto.

The input detection unit 300 may generate detection data for a detection space in which the input object OBJ is located. The sensing data may be used to generate a touch event by the controller 500 described later.

In embodiments, the input sensing unit 300 is disposed in front of the display unit 100 and may acquire side images (or side image data) of the sensing space.

In one embodiment, the input sensing unit 300 may include a first camera device 311 and a second camera device 312. The first camera device 311 is disposed in front of the display unit 100, and is disposed toward the first gaze direction DR1 crossing the display surface of the display unit 100, and generates a first side image for the sensing space can do. The first line-of-sight direction DR1 may be a direction that traverses the display surface in a diagonal direction (ie, a direction inclined based on a side of the display surface), for example, corners (or corners) of the display unit 100 It may be a direction toward the center of the area of the display surface from the edge of one of the (budle).

Similarly, the second camera device 312 is arranged to be spaced apart from the first camera device 311 in front of the display unit 100, but toward the second gaze direction DR2 crossing the display surface of the display unit 100 The second side image of the sensing space may be generated. Here, the second gaze direction DR2 intersects the first gaze direction DR1, and, for example, the second gaze direction DR2 is perpendicular to the first gaze direction DR1 or may form an obtuse angle. . The second line-of-sight direction DR2 may be a direction that crosses the display surface in an oblique direction, and may be, for example, a direction from the other edge of the display unit 100 toward the center of the area of the display surface.

As illustrated in FIG. 1, the first camera device 311 is disposed at the upper left corner of the display unit 100, and the gaze axis of the first camera device 311 may be arranged to face the center of the area of the display surface. have. Similarly, the second camera device 312 is disposed at the upper right corner of the display unit 100, and the gaze axis of the second camera device 312 may be disposed to face the center of the area of the display surface.

The first and second camera devices 311 and 312 may have a viewing angle (or field of view) of 90 degrees or more. In this case, each of the first and second camera devices 311 and 312 may photograph the sensing space in front of the display unit 100 without a shaded area.

For reference, a general camera device used to detect an input object OBJ based on an image (or vision) is disposed toward one side of the display device from the side of the general display device toward the front of the display device, and the front image for the comparison sensing space To acquire. Here, the comparison sensing space, which is a space corresponding to the sensing space according to the embodiment, is spaced apart from the surface of the display unit 100 by a distance of at least a minimum focal length of the camera device. That is, according to the focal length of the lens of the general camera device, the user's hand must be at least 20 cm away from the general camera device to accurately detect the user's hand, so the input object (OBJ) is located in a space located closer than that. It is difficult to detect. To overcome this limitation, a camera device having a short lens focal length can be used, but the camera device has a narrow angle of view, making it difficult to detect an input object OBJ across the entire display device.

On the other hand, since the input sensing units 300 according to the embodiments are arranged to be spaced apart from each other and acquire positional images by acquiring side images, not frontal images of the sensing space, space adjacent thereto from the surface of the display unit 100 ( For example, all input objects OBJ located in the space within 1 m from the display surface of the display unit 100 may be detected.

Meanwhile, in FIG. 1, the first and second camera devices 311 and 312 are illustrated as being disposed at the upper left corner and the upper right corner of the display unit 100, but this is exemplary and is not limited thereto.

In addition, in FIG. 1, the input sensing unit 300 is illustrated as including two camera devices 311 and 312, but this is exemplary, and the input sensing unit 300 may include three or more camera devices. It might be. This will be described later with reference to FIG. 12.

In one embodiment, the input sensing unit 300 may further include an infrared light source 320. The infrared light source 320 may project infrared light into the sensing space. Here, the infrared light has a wavelength of about 940nm, and can be distinguished from the infrared light having a wavelength of 850nm used in surveillance cameras. The infrared light source 320 may include infrared LEDs having a relatively wide projection angle, or infrared laser light sources having a relatively high projection density.

When the infrared light source 320 is provided, the first camera device 311 and the second camera device 312 may be implemented as an infrared camera or include a visible light blocking filter that blocks visible light. Accordingly, sensitivity to side images is improved, and the input object OBJ can be more easily detected by infrared light reflected by the input object OBJ. On the other hand, the display device 1 is applied to an entertainment device such as a slot machine, and can be used in an entertainment area such as a casino. The input object (OBJ) is processed only by processing a general visible image by lighting and images of high luminance of the entertainment area. May not be easily detected. Accordingly, the input sensing unit 300 according to the embodiment uses an infrared light different from the visible light (also, an infrared light having a wavelength different from that used in a surveillance camera in an amusement park), thereby allowing an input object (OBJ). ) Can be detected more accurately.

As illustrated in FIG. 1, the infrared light source 320 may be disposed adjacent to an upper side of the display unit 100. However, this is exemplary, and the infrared light source 320 may be disposed adjacent to the lower side of the display unit 100, or may be disposed on each of the upper side and the lower side of the display unit 100. In addition, the infrared light source 320 may be disposed on the left side or right side of the display unit 100.

The control unit 500 detects the input object OBJ from the side images (ie, the sensed data acquired by the input detection unit 300), and inputs it based on the location information of the input object OBJ in the side images The 3D position information of the object OBJ is calculated, and a spatial touch input may be recognized or a touch event may be generated based on the 3D position information. For example, the controller 500 detects an input object by processing the side images, and sets the location information of each of the preset camera devices 300: 311 and 312 and the location information of the input object in the side images (or 3D position information of the input object may be calculated based on the direction information, and a touch event may be generated based on the change of the 3D position information (for example, a change in the third direction D3).

In addition, the control unit 500 provides image data to the display unit 100 and may control or change image data in response to a touch event. Furthermore, the control unit 500 may generate a sensory control signal in response to a touch event.

In some embodiments, the display device 1 may further include a sensory providing unit 700. The sensory providing unit 700 may provide a specific sensory (eg, tactile) to the input object OBJ.

The sensory providing unit 700 may be disposed in front of the display unit 100. As shown in FIG. 1, the sensory providing unit 700 is disposed adjacent to the lower side of the display unit 100 and may provide a sensory signal to the sensing space. For example, the sensory providing unit 700 includes ultrasonic generators (or ultrasonic transducers), and the ultrasonic generators emit ultrasonic waves from the lower side of the display unit 100 toward the upper side of the display unit 100. You can.

When the input object OBJ enters within a sensory region (eg, an area affected by ultrasound, a space similar to or included in the sensing space), the end of the input object OBJ (ie, within the sensory area) by ultrasound The entered part) may vibrate, and the user may recognize that a touch event is generated based on the vibration of the input object OBJ (eg, vibration of the hand).

The sensory providing unit 700 may generate ultrasonic waves based on the sensory control signal provided from the control unit 500. That is, the sensor providing unit 700 may generate ultrasound only when a touch event occurs. Alternatively, the sensory providing unit 7000 may periodically generate ultrasonic waves regardless of the sensory control signal.

In one embodiment, the sensory providing unit 700 may change the intensity of the ultrasound (ie, the size of the tactile sense) based on the distance between the input object OBJ and the display unit 100. For example, the sensory providing unit 700 may increase the intensity of ultrasound as the input object OBJ is adjacent to the display unit 100.

As described with reference to FIG. 1, the display device 1 acquires side images of the sensing space through the input sensing unit 300 and extracts an input object OBJ from the side images through the control unit 500. And, based on the three-dimensional position of the input object (OBJ) may recognize a spatial touch input or generate a touch event.

In addition, the input sensing unit 300 detects the input object OBJ adjacent to the display surface of the display unit 100 by acquiring side images of the sensing space, and in the third direction D3 of the input object OBJ The position change (or the distance from the display surface to the input object OBJ) can be more accurately detected.

Furthermore, the first and second camera devices 311 and 312 included in the input sensing unit 300 are spaced apart from each other, and are disposed adjacent to different corners of the display unit 100, respectively, so that the display surface is not shaded. It is possible to detect the input object (OBJ) in the detection space of the entire front.

Meanwhile, the input sensing unit 300 and the control unit 500 (and the sensory providing unit 700) may be implemented as one module (eg, a spatial touch sensing module).

2A and 2B are exploded perspective views illustrating an example of a display unit included in the display device of FIG. 1. 2C is a bottom side view illustrating an example of the display device of FIG. 1. FIG. 2D is an enlarged view of the area Q1 in FIG. 1.

2A to 2D, the display unit 100 may include a display module 110 and a housing 130: 131, 132, 133, and 134.

The display module 110 is implemented as a liquid crystal display module, and as illustrated in FIG. 2B, may include a liquid crystal display panel 111, a light source 113, a back plate 115 and a diffusion sheet 117.

The liquid crystal display panel 111 may be a curved display panel that is concavely curved left and right based on a vertical axis. The liquid crystal display panel 111 may include a pixel electrode and a common electrode that face each other, a liquid crystal layer (or liquid crystal cells) disposed between them, transistors that transmit voltage to the pixel electrode, color filters, and polarizing members. .

The light source 113 may provide light to the liquid crystal display panel 111. The light source 113 includes a plurality of LEDs, and the LEDs may be attached to the back plate 115 in a matrix form. In FIG. 2, it is illustrated that the light source 113 is arranged in a direct shape, but the light source 113 may be arranged in an edge shape.

The back plate 115 supports the light source 113 and covers the liquid crystal display panel 111. The back plate 115 follows the circumferential direction of the liquid crystal display panel 111 on a concentric circle of the same origin as the origin (ie, the center of the virtual circle obtained by extending the curved surface) of the curved liquid crystal display panel 111 in a curved shape. It can have a set width. The back plate 115 may include a plurality of rectangular plates, and each plate may be disposed to face the origin of the liquid crystal display panel 111.

The optical sheet 117 is disposed between the liquid crystal display panel 111 and the light source 113. The optical sheet 117 may include a diffusion sheet, a prism sheet, a brightness enhancing sheet, and the like.

Meanwhile, although not illustrated, the display module 110 may further include a window disposed in front of the liquid crystal display panel 111 to protect the liquid crystal display panel 111.

Referring to FIG. 2A again, the housing 130 may accommodate the display module 110 and protect the display module 110 from the rear and side surfaces of the display module 110. In addition, the housing 130 accommodates the input sensing unit 300 and / or sensory providing unit 700, or the input sensing unit 300 and / or sensory providing unit 700 is mounted in the housing 130. ).

The housing 130 may include a body part 131, an upper cover 132, a side cover 133, and a support part 134.

The body part 131 may have a rectangular planar shape corresponding to the display module 110, and the display device 1 may be curved together in a bending direction, and may include a storage space formed by recessing the front surface. The display module 110 may be stored in the storage space.

At least a portion of the rear surface of the body portion 131 may be cut or opened, and a module-shaped control unit 150 may be disposed in the cut or open portion of the body portion 131.

The upper cover 132 is disposed on an upper side (or an upper front side) of the body portion 131, and may include an empty space formed between the body portion 131 therein. The input sensing unit 300 may be disposed in the empty space. The upper cover 132 and the input sensing unit 300 are configured as one module, and may be disposed or combined on the upper side of the body unit 131.

As illustrated in FIG. 2C, an opening OP communicating between the empty space and the outside may be formed between the upper cover 132 and the main body 131 under the upper cover 132. The housing 130 may further include a transparent plate 132-1 covering at least a portion of the opening. The upper cover 132 is made of an opaque and rigid material, so that the input sensing unit 300 can be prevented from being exposed to the outside. Infrared light emitted from the input sensing unit 300 (or the infrared light source 320) may be transmitted through the transparent plate 132-1 disposed in the opening OP to be provided to the sensing space.

Meanwhile, a part of the opening OP of the upper cover 132 is not covered by the transparent plate 132-1, and the first and second camera devices 311 and 312 (for example, camera devices ( 311, 312) each lens) may be exposed to the outside. With this structure, it is possible to prevent the infrared light reflected by the input object OBJ from being totally reflected by the transparent plate 132-1 (or the refractive index of the transparent plate 132-1).

Referring to FIG. 2A again, the side cover 133 has a rectangular planar shape and may be disposed on the side of the body portion 131. The side cover 133 may block light (eg, infrared light) flowing from the side surface of the display module 110. In addition, the side cover 133 may provide an installation area of the input sensing unit 300 (or first and second camera devices 321 and 322).

In one embodiment, the side cover 133 may define a sensing space. For example, in the side images obtained from the first and second camera devices 321 and 322, it may be determined that only the input object OBJ displayed overlapping the side cover 133 exists in the detection area. have.

The support part 134 (or a support member) may support the first and second camera devices 311 and 312 and the infrared light source 320. The support 134 may include a first support 134-1 supporting the first and second camera devices 311 and 312 and a second support (not shown) supporting the infrared light source 320. .

As shown in FIG. 2D, the first support part 134-1 is fixed to the upper surface of the main body part 131 (or the display module 110) and the fixing part 134a and the fixing part 134a. A first extension 134b extending along a diagonal direction (ie, a fourth direction D4, a direction perpendicular to the line of sight of the first camera device 311, or a direction forming an acute angle with the fixing part 134a) ) And a second extension 134c extending forward of the display module 110 from the first extension 134b. The first camera device 311 (or the second camera device 312) may be fixed to the inner surface of the second extension part 134c.

The horizontal angle of the first camera device 311 (that is, the horizontal component of the gaze axis of the first camera device 311) is adjusted according to the alignment of the fixing part 134a and the body part 131, and the fixing part 134a The vertical angle of the first camera device 311 (ie, the vertical component of the gaze axis of the first camera device 311) may be adjusted according to the bent angle of the first extension part 134b.

Meanwhile, in FIG. 2D, the first support portion 134-1 is illustrated as being disposed on the upper side of the body portion 131, but is not limited thereto, and the first support portion 134-1 is the body portion 131. It may be disposed on the side or side cover 133 of.

The second support part (not shown) may be formed to extend upward from the upper surface of the main body part 131 (or the display module 110) and bend toward the front of the display module 110. The second support portion has an “a” shape (or an inverted “L” shape) and an infrared light source on the inner side of the second support portion (not shown) (ie, the side facing the upper side of the main body portion 131). 320 may be disposed. The second support prevents the infrared light emitted from the infrared light source 320 from traveling in the horizontal direction (that is, the infrared light is directly introduced into the first and second camera devices 311 and 312). It may also include a side light blocking member.

In some embodiments, the display unit 100 may further include a three-dimensional filter unit (not shown). The three-dimensional filter unit may be disposed in front of the display module 110. For example, the three-dimensional filter unit may have a radius of curvature that is substantially the same as or similar to a radius of curvature of the display module 110 and may be disposed in front of the display module 110. The three-dimensional filter unit may be in close contact with the front surface of the display module 110 or may be disposed spaced apart from the front surface of the display module 110.

The three-dimensional filter unit (not shown) may include a lenticular lens sheet. By providing the left-eye image of the display module 110 to the left eye of the user and the right-eye image of the display module 110 by the lenticular lens, a 3D stereoscopic image may be provided to the user. When the lenticular lens is concavely formed with the same curvature as the display module 110, an optical illusion phenomenon such as the presence of an image in the air that is a concave space may occur. Accordingly, it appears that the image is displayed on the space due to the optical illusion, and the user can maximize the sense of space by touching the image with his or her hands at a certain depth of the concave space.

As described with reference to FIGS. 2A to 2D, the display unit 100 covers the input sensing unit 300 using the upper cover 132, but at least a portion of the upper cover 132 includes the input sensing unit 300 By exposing the first and second camera devices 311 and 312 of, side images that reflect all of the light reflected by the input object OBJ may be generated. In addition, the display unit 100 includes side covers 133a and 133b disposed on the side of the display module 110 to block light from outside, and to more easily define or form a sensing space. You can.

3A to 3C are diagrams illustrating an example of a sensing space formed by an input sensing unit included in the display device of FIG. 1. 3B and 3C show a plane of the sensing space.

Referring first to FIGS. 3A and 3B, the first and second camera devices 311 and 312 are disposed in front of the display unit 110, as described above, in the upper left corner and the upper right side of the display unit 110. It can be arranged at each corner. Also, the first and second camera devices 311 and 312 may be disposed such that the line of sight of the first and second camera devices 311 and 312 faces the center of the area of the display surface. The first and second camera devices 311 and 312 may have an angle of view of 90 degrees or more, for example, an angle of view of 90 degrees or 140 degrees.

The sensing space SA may be formed in an area where the photographing area of the first camera device 311 overlaps the photographing area of the second camera device 312. As the sensing space SA is adjacent to the first and second camera devices 311 and 312, the width (or thickness, that is, the length in a direction perpendicular to the display surface of the display unit 110) is smaller. The distance from the first and second camera devices 311 and 312 may increase. However, the present invention is not limited thereto, and the shape and width of the sensing space may vary according to the curvature of the display unit 110 and the arrangement directions of the first and second camera devices 311 and 312.

As illustrated in FIG. 3B, the sensing space SA may have a circular (or fan-shaped) planar shape based on a point (see 'P0') located in front of the display device 1.

In one embodiment, the edge of the viewing angle of the first camera device 311 and the edge of the viewing angle of the second camera device 312 are the origin (P0) of the display device 1 (or the display surface of the display unit 100). ), Or the origin P0 of the display device 1 may be located within the field of view of the first camera device 311 and the field of view of the second camera device 312. In this case, the sensing space SA has the same thickness (that is, the length in a direction perpendicular to the display surface) on the entire front surface of the display device 1, and according to the shape of the sensing space, the front surface of the display device 1 An input range of a spatial touch having the same maximum depth may be provided. For example, when the display device 1 has a 55-inch display surface and a radius of curvature of about 1.5 m, the first and second camera devices 311 and 312 may also have an area center (' P1 '), the edges of the viewing angles of the first and second camera devices 311 and 312 may pass the origin P0 of the display device 1. In this case, the sensing space SA has an arc shape having a radius of about 1.5 m, and the display device 1 is based on a user (or a user's gaze) located at the origin P0 of the display device 1 Thus, an input range of a spatial touch having the same depth DE1 of up to 1.5m in the entire area of the front surface of the display device 1 may be provided.

On the other hand, when the infrared light source 320 projects infrared light to an area having a constant width (for example, 30 cm) in front of the display device 1, the display device 1 is front as shown in FIG. 3C. A sensing space SA having the same depth DE2 (eg, a maximum depth of 30 cm) may be formed in the entire area.

As described with reference to FIGS. 3A to 3C, when the viewing angles of the first and second camera devices 311 and 312 are arranged to pass through the origin P0 of the radius of curvature of the display device 1, separately Without the image processing of, the sensing space SA having the same depth ('DE2' in FIG. 3C) is formed on the entire front surface of the display device 1, or the same maximum depth on the entire front surface of the display device 1 (FIG. 3B) An input range of a spatial touch having 'DE1') may be provided.

4 is a diagram illustrating a display device according to some example embodiments of the present invention. 5 is a plan view illustrating an example of a sensing space formed by the display device of FIG. 4.

1, 4 and 5, the display device 1_6 is different from the display device 1 of FIG. 1 in that it includes a flat display unit 100. Except for the flat display unit 100 (and the components corresponding to the flat display unit 100), the display device 1_6 in FIG. 4 is substantially the same or similar to the display device 1 in FIG. 1, and thus, overlaps with the description Will not repeat.

The display device 1_6 includes first and second camera devices 311 and 312, and the first camera device 311 is disposed adjacent to the upper left corner of the display unit 100, and the second camera device ( 312) may be disposed adjacent to the upper right corner of the display unit 100.

However, the first camera device 311 is disposed adjacent to the end of the first side cover 133a (or the left side cover) coupled to the left side of the display unit 100, and the second camera device 312 is The second side cover 133b (or the right side cover) coupled to the right side of the display unit 100 may be disposed adjacent to the end. That is, the first and second camera devices 311 and 312 are the widths of the first and second side covers 133a and 133b (ie, perpendicular to the display surface) from the display unit 100 (or the display surface). Direction).

As the display unit 100 is implemented in a planar shape rather than a curved shape, when the first and second camera devices 311 and 312 are disposed adjacent to the display surface based on FIG. 5B, at both sides of the display unit 100 Since the thickness of the sensing space is relatively small, a spatial touch input having a sufficient depth may not be provided at both sides of the display unit 100. Alternatively, when the first and second camera devices 311 and 312 are disposed toward the front, the sensing space SA is unnecessarily large and unnecessary information (for example, objects spaced 1 m or more from the display surface) By detecting this, the load of the display device can be increased.

Accordingly, in consideration of the depth of the spatial touch input to be provided on both sides of the display unit 100, the first and second camera devices 311 and 312 are spaced forward from the display unit 100, so that the first and first 2 may be disposed on the side covers 133a and 133b, respectively. For example, as the depth of the spatial touch input to be provided is greater, the first and second camera devices 311 and 312 may be disposed more spaced apart from the display unit 100, and also, the first and second side surfaces. The widths of the covers 133a and 133b (ie, the length in a direction perpendicular to the display surface) can also be increased.

Meanwhile, when the first and second camera devices 311 and 312 are spaced apart from the display unit 100, shaded areas may occur on both sides of the display unit 100. Accordingly, the first and second camera devices 311 and 312 may be arranged to face a reference point located behind the display surface rather than the center of the area of the display surface.

6 is a block diagram illustrating an example of a control unit included in the display device of FIG. 1. 7A to 7F are diagrams illustrating a process of detecting an input object in the control unit of FIG. 6.

7A is an example of a pointer that is an input object, and a user's hand located in the sensing space of the display device 1 is exemplarily illustrated, and FIG. 7B is a first camera device included in the display device 1 of FIG. 7A. The first side image obtained at 311 is shown, and FIG. 7C shows the second side image obtained at the second camera device 312 included in the display device 1 of FIG. 7A. Infrared light (ie, infrared light projected from the infrared light source 320 described with reference to FIG. 1 to the sensing space) is reflected or scattered by the user's hand located in the sensing space, so that the user's hand may appear relatively bright. have.

Referring to FIG. 6, the control unit 500 may include a pre-processing unit 610, an extraction unit 620, and a calculation unit 630. In addition, the control unit 500 may further include an event generation unit 640. Since the image processing processes for the first and second side images IMGAE1 and IMAGE2 are substantially the same or similar to each other, the image processing processes for the first and second side images IMGAE1 and IMAGE2 are common to each other. The process will be described based on the first side image IMAGE1.

The pre-processing unit 610 may remove the background image from the first side image IMAGE1, remove noise, and also remove the shadow of the input object. That is, the pre-processing unit 610 may pre-process the first side image so that the hand image can be easily extracted from the first side image IMAGE1.

The pre-processing unit 610 may include a background removal unit 611, a noise removal unit 612, and a shadow removal unit 613.

The background removing unit 611 may generate or obtain a first pre-processed image IMAGE_B1 by differentially calculating a preset background image IMAGE_BACK from the first side image IMAGE1. Here, the preset background image IMAGE_BACK may be an image of an area in which only the display surface of the display unit 100 is displayed. The background removing unit 611 may convert the pixel value of the corresponding area to 0 (for example, black).

The noise removing unit 612 may sample (or binarize) and smooth the first pre-processed image IMAGE_B1 to obtain a first original image IMAEG_N1 from which noise is removed. For example, the noise removing unit 612 may replace the pixel values in the first pre-processed image IMAGE_B1 with 0 (for example, black) or 255 (for example, white) based on the threshold value. .

The shadow removal unit 613 may apply a shadow removal algorithm to the first original image IMAGE_N1 to remove the shadow image and generate a shadow removal image IMAGE_S1. Here, the shadow image may be an image displayed by reflecting the hand by the display surface of the display unit 100.

As illustrated in FIG. 7D, the first original image IMAGE_N1 may include candidate object images OBJ_1 and OBJ_1S. Here, the candidate object images OBJ_1 and OBJ_1S are images that can be recognized as an input object OBJ, and may have a size larger than a reference size (or a reference area). In this case, the shadow removing unit 613 determines the shadow image OBJ_1S among the candidate object images OBJ_1 and OBJ_1S based on the size, symmetry, and placement position of the candidate object images OBJ_1 and OBJ_1S, and 1 The shadow image OBJ_1S may be removed from the original image IMAGE_N1. For example, when candidate object images OBJ_1 and OBJ_1S of the same or similar shape exist, the shadow image OBJ_1S having a relatively small size and adjacent to the display unit 100 may be selected and removed.

The extraction unit 620 may extract the hand image from the shadow removal image IMAGE_S1 finally output from the pre-processing unit 610. The extraction unit 620 may include a detection unit 621, a tracking unit 622, and a correction unit 623.

The detector 621 may match the preset patterns (for example, object patterns including features or characteristic points of the input object) with respect to the shadow removal image IMAGE_S1 to detect the pointer image. For example, the detection unit 621 extracts the candidate object image OBJ1 having a preset size (ie, pixels having a predetermined number or more), and matches the candidate object image OBJ1 with a preset pattern to generate a hand image. Can be detected. Here, the preset patterns are, for example, a pattern corresponding to a hand in which only one finger is extended (a pattern including feature points up to a finger, a palm, and a wrist), a pattern corresponding to a fist in a fist, and all fingers extended It may include a pattern corresponding to the hand.

In addition, the detection unit 621 of the extraction unit 620, when the image of the pointer (hand image in the drawing) as an input object is detected, the corresponding pattern (that is, the pattern used to detect the hand image, or the corresponding pattern The reference point of the input object image may be determined based on the previously defined feature point. For example, as illustrated in FIG. 7D, when a hand image OBJ_1 corresponding to a pattern indicating a state in which one finger is stretched is detected, a point at the end of the finger may be determined as the reference point P_OBJ1. For another example, when a hand image corresponding to a pattern indicating a state in which all fingers are stretched is detected, the center point of the palm may be determined as the reference point P_OBJ1.

Meanwhile, the shape of the input image may be changed according to a distance between the first and second camera devices 311 and 312 for a pointer such as a finger or a pen. In this way, even when the input image shape is changed, the detector 621 may further perform shape analysis, ratio (size) analysis, and / or multi-camera weight analysis to accurately recognize and detect the pointer. To this end, the detection unit 621 may further include a shape analysis unit, a ratio (size) analysis unit, and / or a multi-camera weight analysis unit. 8 to 10 are referred to for a detailed description.

8A and 8B are schematic views showing an input image of a pointer according to a position.

8A and 8B, an image image shape of a pointer input to the camera device 310 varies according to a distance from the camera device 310 to the pointer. That is, in the case of the user's hand, as shown in FIG. 8A, the shape of the entire hand may be input as an image image at a distance, but at a short distance, the shape of the hand may be recognized as a single shape as a partial shape. In addition, in the case of a pen, as shown in FIG. 8B, an input image of the shape of the entire pen can be obtained at a long distance, but at a short distance, the shape of the pen may be partially input and viewed as a single mass.

In this way, even when the pointer is located at a short distance, it is possible to identify where the pointer points on the screen by performing shape analysis. 9 is a schematic diagram showing a comparison of an original image and a converted image of a proximity pointer. As shown in FIG. 9, when the feature points are extracted and connected to the original image at a short distance, the ends of the hand or pen have a relatively small curvature compared to the surroundings. By recognizing this curvature, it is possible to determine where the pointer is pointing in the near partial input image. The pointer end of the short-distance input image may be determined as a point having the same curvature compared to the curvature of the previously stored image pattern, or may be determined as a point having a minimum curvature in the curve formed by the feature points.

10 is a schematic diagram showing a method of analyzing a ratio of a converted image. Referring to FIG. 10, in the case of a far image, ratio analysis may be performed to distinguish the pointer from the noise image. Ratio analysis may be performed by comparing the ratio of the width of the image in the vertical direction (the width of the top and bottom) and the width of the horizontal direction (the width of the left and right). For example, in the case of a hand or pen, the width in the horizontal direction is greater than the width in the vertical direction, whereas in the case of a human body, the width in the vertical direction may be greater than the width in the horizontal direction. As another example, the ratio analysis may be performed by comparing the ratio of the first width in the longest direction to the second width in the direction intersecting (usually vertical) to the first width. In the illustrated example, the ratio of the first width to the second width of the hand or pen is greater than or equal to 2: 1, but the human body can be divided into a ratio of less than 2: 1 between the first width and the second width. As described above, by comparing the ratio of the width in the vertical direction and the width in the horizontal direction or the ratio between the first width and the second width, the pointer can be accurately recognized and a noise image can be removed.

Meanwhile, an input image in which one pointer is input to each camera device 311 and 312 is different according to a distance. For accurate object recognition, different weights may be assigned for each distance from the camera devices 311 and 312. As described above, even a partial input image at a short distance may determine the end of the pointer and the direction in which the pointer is directed through shape analysis, but its accuracy may be deteriorated compared to the entire remote input image. In order to compensate for this, the accuracy of object recognition may be improved by further adding a weight to a remote input image among a plurality of input images for one pointer. For example, in the display device 1 illustrated in FIG. 1, when the input object OBJ is positioned relatively closer to the first camera device 311 than the second camera device 312, the second camera device 312 ), Object recognition may be performed by assigning more weight to the input image image. The weight can be adjusted from a ratio of 100: 0 to a ratio of 50:50 depending on the distance away. The distance away from the camera devices 311 and 312 may be calculated by measuring the size of the image of the pointer or by receiving feedback of the three-dimensional position of the pointer calculated by the calculation unit 630 described later.

Referring back to FIGS. 6 and 7, the tracking unit 622 may track the pointer image over time using a pointer (hand in the drawing) tracking algorithm as a target object. For example, the tracking unit 622 may predict the next position of the hand image based on the movement path of the hand image (or the reference point) over time, and the hand image based on the next position of the predicted hand image. Even if it is detected quickly or the hand image is not temporarily detected, the hand image (or reference point) can be maintained for a specific time. Through this method, it is possible to prevent the touch input from being made because the hand image is not recognized temporarily.

The correction unit 623 may remove noise over time for a change in the position of the reference point P_OBJ1. For example, when the pointer to be tracked moves rapidly or the recognized reference point (P_OBJ1) is shaking, the correction unit 623 uses a filter such as a Kalman filter or a low pass filter to position the reference point (P_OBJ1). Can be corrected.

That is, the extractor 620 detects the pointer image (and / or the reference point of the pointer image) from the pre-processed side images, but can track and correct the pointer image to be suitable for use as a spatial touch input.

The calculator 630 may calculate a 3D position of the pointer in the sensing space based on the pointer image (or the reference point P_OBJ1). The calculator 630 may include a first position calculator 631 and a second position calculator 632.

The first position calculator 631 may calculate a two-dimensional position (or coordinate) of the pointer in a plane corresponding to the display surface of the display unit 100. As illustrated in FIG. 7A, the first position calculator 631 may calculate the eleventh position angle ANG11 of the hand image OBJ_1 based on the reference point of the hand image OBJ_1. Here, the eleventh position angle ANG11 may be an angle between the input object OBJ from the upper side of the sensing space with respect to the first camera device 131. The first position calculator 631 may calculate the eleventh position angle ANG11 based on the distance (or the number of pixels) between the reference points of the hand image OBJ_1 from the upper side of the first side image. Similarly, as illustrated in FIG. 7C, the first position calculator 631 may calculate the 21st position angle ANG21 of the hand image OBJ_2 based on the reference point of the hand image OBJ_2.

Subsequently, as illustrated in FIG. 7E, the first position calculator 631 includes first camera position information P_C1 of the first camera device 311 and second camera positions of the second camera device 312. Based on the information P_C2, the eleventh position angle ANG11 and the twenty-first position angle ANG21, the two-dimensional position P_OBJ_XY of the hand as a pointer can be calculated. The two-dimensional position of the hand (P_OBJ_XY) may be a point (or coordinate) where the hand is positioned on the display surface of the display unit 100.

The second position calculator 632 may calculate the depth of the pointer (or the depth value and the distance the hand is separated from the display surface) based on the display surface of the display unit 100. 7A to 7C, the second position calculator 632 calculates the twelfth position angle ANG12 and the twenty-second position angle ANG22 based on the reference points of the hand images OBJ_1 and OBJ_2. You can. Here, the twelfth position angle ANG12 may be an angle between the input object OBJ from the front of the sensing space (or the display surface of the display unit 100) based on the first camera device 131.

Then, the second position calculator 632, as shown in Figure 7f, the first camera position information of the first camera device 311 (P_C1), the second camera of the second camera device 312 Based on the position information P_C2, the twelfth position angle ANG12, and the twenty-second position angle ANG22, the depth P_OBJ_Z of the hand as a pointer can be calculated.

Referring to FIG. 6 again, the second position calculator 632 calculates the three-dimensional position of the hand based on the two-dimensional position P_OBJ_XY of the hand and the depth P_OBJ_Z of the hand calculated by the first position calculator 631. (P_XYZ).

Meanwhile, the event generator 640 may generate a touch event based on the three-dimensional position P_XYZ of the pointer (hand) generated by the calculator 630.

As described with reference to FIGS. 6 to 7F, the controller 600 detects a pointer (or hand) as an input object from side images through image processing, calculates a three-dimensional position of the pointer, and three-dimensional of the pointer A touch event may be generated based on the location.

In addition, the control unit 600 may detect the pointer more accurately by removing the shadow reflected by the display surface of the display unit 100.

Furthermore, the controller 600 can generate a more accurate touch event by tracking the pointer image and correcting the pointer image over time.

11A and 11B are views illustrating an example of a sensory providing unit included in the display device of FIG. 1.

Referring to FIGS. 1 and 11A, the sensory providing unit 700 is disposed on the lower front side of the display unit 100, and may include a plurality of ultrasonic transducers 710 arranged along the first direction D1. have. The ultrasonic transducer 710 may generate ultrasonic waves having directivity in the second direction D2 (for example, ultrasonic waves having a frequency of 40 kHZ), and apply pressure to at least a portion of the input object OBJ. That is, the sensory providing unit 700 may provide a tactile feel to the input object OBJ using the law of acoustic radiation force.

In one embodiment, the sensory providing unit 700 may include a first ultrasound transducer 711 and a second ultrasound transducer 712 having different frequencies or different intensities. The first ultrasonic transducer 711 and the second ultrasonic transducer 712 may be arranged as a plurality of rows or columns. For example, the first ultrasound transducer 711 and the second ultrasound transducer 712 each form one row and are arranged along the first direction D1, and the second ultrasound transducer 712 is the first. The ultrasound transducer 712 may be disposed adjacent to the display unit 100.

The second ultrasonic transducer 712 may have a lower frequency or greater intensity than the first ultrasonic transducer 711. In this case, as the input object OBJ is adjacent to the display unit 100, more pressure (or tactile feel) may be provided to the input object OBJ.

In FIG. 11A, the ultrasonic transducer 710 is illustrated to emit ultrasonic waves in the second direction D2, but is not limited thereto.

Referring to FIG. 11B, the sensory providing unit 700_1 may include third ultrasonic transducers 713 and fourth ultrasonic transducers 714. The third ultrasonic transducers 713 and the fourth ultrasonic transducers 714 may generate and output ultrasonic waves having the same frequency (eg, 40 KHz).

The third ultrasonic transducers 713 are disposed adjacent to the left side from the front lower side of the display unit 100, and are disposed to face the fifth direction D5 to emit ultrasonic waves toward the fifth direction D5. You can. Similarly, the fourth ultrasonic transducers 714 are disposed adjacent to the right side from the front lower side of the display unit 100, and are disposed to face the sixth direction D6, so as to move toward the sixth direction D6 Can radiate. As the sixth direction D6 intersects with the fifth direction D5, ultrasonic waves output from the third ultrasonic transducers 713 include ultrasonic waves output from the fourth ultrasonic transducers 714 and a plurality of points. You can cross in the field.

In this case, the controller 500 determines a specific point (that is, a point to provide tactile sense) based on the three-dimensional position of the input object OBJ, and the third and fourth ultrasonic transducers based on the determined point Some of the operations 713 and 714 may be operated, or a sensory control signal that operates them may be generated. For example, in order to provide a tactile feel to the input object OBJ shown in FIG. 11B, the controller 700 may operate the first sub-ultrasonic transducer 713a and the second sub-ultrasonic transducer 714a. . At this time, a pressure sufficient to be recognized as a tactile sense may be provided to the user only at the point (or area) where the ultrasonic waves overlap. That is, the user may be touched only at the point where the ultrasonic waves overlap.

Meanwhile, in FIG. 11B, the third ultrasound transducers 713 and the fourth ultrasound transducers 714 are illustrated to be arranged separately from the lower side of the display unit 100, but this is exemplary and is not limited thereto. no. For example, the third and fourth ultrasonic transducers 713 and 714 may be alternately arranged. For another example, the third ultrasonic transducers 713 are disposed under the display unit 100, and the fourth ultrasonic transducers 714 are disposed at the left (or right) side of the display unit 100, They may emit ultrasonic waves in mutually intersecting directions. That is, if the third and fourth ultrasonic transducers 713 and 714 emit ultrasonic waves that cross each other at a plurality of points, their positions are not limited.

Also, the third and fourth ultrasonic transducers 713 and 714 may be arranged along a plurality of columns (or a plurality of rows), similar to the sensory providing unit 700 illustrated in FIG. 11A.

12 is a view showing a display device according to another exemplary embodiment of the present invention. 13 is a diagram illustrating a configuration in which the display device of FIG. 12 detects multi-input. 13 shows the positions of the camera devices 311 to 314 and the input objects OBJ1 and OBJ2 on the front surface of the display device.

Referring to FIG. 12, the display device 1_7 is different from the display device 1 of FIG. 1 in that it further includes third and fourth camera devices 313 and 314.

The third camera device 313 is arranged to be spaced apart from the first and second camera devices 311 and 312 in front of the display unit 100, and the third gaze direction DR3 crossing the display surface of the display unit 100 ) To generate a third side image of the sensing space. Similarly, the fourth camera device 314 is disposed away from the third camera device 313 (and the first and second camera devices 311 and 312) in front of the display unit 100, and the display device ( Arranged toward the fourth gaze direction DR4 traversing the display surface of 1), a fourth side image of the sensing space may be generated.

The third and fourth camera devices 313 and 314 may be provided in a lower cover substantially the same or similar to the upper cover 132 described with reference to FIG. 2A. In addition, the third and fourth camera devices 313 and 314 may be fixed by the support 134 described with reference to FIG. 2D.

The display device 1_7 may detect two input objects (eg, two hands of the user) through the first to fourth camera devices 311, 312, 313, and 314.

When the first and second input objects OBJ1 and OBJ2 are located in the sensing space in front of the display device 1_7, one of the first and second camera devices 311 and 312 is the first and second An image of one of the input objects OBJ1 and OBJ2 may not be acquired.

That is, as illustrated in FIG. 12, when the first input object OBJ1 and the second input object OBJ2 are positioned along the first gaze direction DRR1 (or the third gaze direction DRR3), Since the second input object OBJ2 is covered by the first input object OBJ1 based on the first camera device 311, the first camera device 311 generates an image for the second input object OBJ2. You may not be able to. As the first input object OBJ1 has a volume rather than a dot, as shown in FIG. 13, a shadow zone SZ for the first camera device 311 may occur.

Accordingly, the display device 1_7 further includes third and fourth camera devices 313 and 314, and the third input device OBJ1 as well as the third and fourth camera devices 313 and 314 are used as well. In addition, the second input object OBJ2 located in the shaded area SZ may be detected.

6 and 13, position information (eg, an eleventh position) of a first reference point P_OBJ1 with respect to a first input object OBJ from a first side image obtained from the first camera device 311 Only angle ANG11) can be obtained. Similarly, only the position information of the second reference point P_OBJ2 for the second input object OBJ2 (for example, the 42th position angle ANG42) from the fourth side image obtained from the fourth camera device 314 Can be obtained. Meanwhile, location information for each of the first and second input objects OBJ1 and OBJ2 may be obtained from the second and third side images obtained from the second and third camera devices 312 and 313. .

In this case, the controller 600 determines a camera device in which only position information for one input object is obtained and a camera device in which position information for two input objects is obtained as a group, and the Based on this, a 3D position for the one input object may be calculated. Also, the other two camera devices may be determined as another group, and a 3D position of the other of the two input objects may be calculated based on the information of the group.

For example, the second camera devices 312 are grouped into a first group together with the first camera device 311, and the first and second camera devices 311 and 312 are the first of the display device 1_7. As it is located on the long side LS1, a three-dimensional position of the first input object OBJ1 adjacent to the first long side LS1 among the first and second input objects may be calculated. Based on the second camera device 312, since the 21st position angle ANG21 of the first input object OBJ1 is smaller than the 22nd position angle ANG22 of the second input object OBJ2, the controller 500 Determines the 21st position angle ANG21 as the position angle of the first input object OBJ1, and 3 of the first input object OBJ1 based on the 11th position angle ANG11 and the 21st position angle ANG21. You can calculate the dimensional position.

Similarly, the third camera device 313 is grouped into a second group together with the fourth camera device 314, and the third and fourth camera devices 313, 314 are the second long side of the display device 1_7. As it is located at (LS2), a three-dimensional position of the second input object OBJ2 adjacent to the second long side LS2 of the first and second input objects may be calculated. Based on the third camera device 313, since the 32nd position angle ANG32 of the second input object OBJ2 is smaller than the 31st position angle ANG31 of the first input object OBJ1, the control unit 500 Determines the 32nd position angle ANG32 as the position angle of the second input object OBJ2, and 3 of the second input object OBJ2 based on the 32nd position angle ANG32 and the 42th position angle ANG42. You can calculate the dimensional position.

For another example, the third camera device 313 is grouped into a first group together with the first camera device 311, and the first and third camera devices 311 and 313 are the first device of the display device 1_7. As adjacent to one short side SS1, a three-dimensional position of the first input object OBJ1 adjacent to the first short side SS1 among the first and second input objects may be calculated. Based on the third camera device 313, since the 31st position angle ANG31 of the first input object OBJ1 is greater than the 32nd position angle ANG32 of the second input object OBJ2, the controller 500 Determines the 31st position angle ANG31 as the position angle of the first input object OBJ1, and 3 of the first input object OBJ1 based on the 11th position angle ANG11 and the 31st position angle ANG31. You can calculate the dimensional position.

Similarly, the second camera device 312 is grouped into a second group together with the fourth camera device 314, and the second and fourth camera devices 312, 314 are the second short sides of the display device 1_7. As it is located at (SS2), a three-dimensional position of the second input object OBJ2 adjacent to the second short side SS2 among the first and second input objects may be calculated. Since the second position angle ANG32 of the second input object OBJ2 is greater than the second position angle ANG31 of the first input object OBJ1 based on the second camera device 312, the controller 500 Determines the 22nd position angle ANG22 as the position angle of the second input object OBJ2, and 3 of the second input object OBJ2 based on the 22nd position angle ANG22 and the 42th position angle ANG42. You can calculate the dimensional position.

That is, the control unit 500 groups the first to fourth camera devices 311, 312, 313, and 314 into first and second groups, and a first input object among side images corresponding to the first group (Or, information of the input object) is obtained to calculate a 3D position of the first input object, and only a second input object among side images corresponding to the second group is obtained to calculate a 3D position of the second input object can do.

As described with reference to FIGS. 12 and 13, the display device 1_7 may set two camera devices as one group to detect one input object for each group, and accordingly, the display device 1_7 Can detect multiple input objects.

Meanwhile, in FIGS. 12 and 13, the display device 1_7 is illustrated as detecting two input objects using four camera devices, but this is exemplary and is not limited thereto. The display device 1_7 may accurately detect N input objects using 2N (where N is an integer of 2 or more) camera devices spaced apart from each other.

14 is a flowchart illustrating an example of a spatial touch sensing method performed in the display device of FIG. 1. 15A is a flowchart illustrating an example of a process in which the method of FIG. 14 preprocesses a side image. 15B is a flowchart illustrating an example of a process in which the method of FIG. 14 detects an input object.

1, 6, 14, and 15, the method of FIG. 14 uses the input sensing unit 300 (or the first and second camera devices 311 and 312) for the detection space. Side images may be acquired (S1110).

Thereafter, the method of FIG. 14 may pre-process side images (S1120). The method of FIG. 14 removes noise by removing a preset background image from side images (S1210), binarizing and smoothing side images (S1220), and including the side images as described with reference to FIG. The shadow image may be removed based on the size and symmetry of the candidate object images (S1230).

Thereafter, the method of FIG. 14 may detect an input object from pre-processed images (ie, pre-processed side images) (S1130). As described with reference to FIG. 6, the method of FIG. 14 detects an object image for an input object through mapping with a preset pattern, and determines a reference point of the input object based on the preset pattern from the detected object image Then, the position of the reference point (eg, position angles) in the pre-processed images may be calculated (S1240).

For example, referring to FIGS. 7A to 7C, the method of FIG. 14 detects a first object image OBJ_1 corresponding to the input object OBJ from the first side image IMAGE1 based on a preset pattern , A first reference point of the first object image OBJ_1 is determined based on a predetermined feature point in a preset pattern, and a first object corresponding to the input object OBJ from the second side image IMAGE2 based on the preset pattern. The second object image OBJ_2 may be detected, and a second reference point of the second object image OBJ_2 may be determined based on a pre-defined feature point in a preset pattern.

In addition, the method of FIG. 14 continuously tracks the position of the input object (or the reference point) in side images over time (S1250), and changes the temporary position of the input object (or the reference point) over time. It can be corrected (S1260).

Thereafter, the method of FIG. 14 may calculate a 3D position of the input object based on the object image (or the reference point) (S1140). As described with reference to FIG. 6, in the method of FIG. 14, predetermined position information of the first and second camera devices 311 and 312 and a reference point of the calculated input object (eg, the position of the input object) Based on the angles), a 3D position of the input object may be calculated.

For example, referring to FIGS. 7A to 7E, the method of FIG. 14 calculates the eleventh position angle ANG11 based on the position of the first reference point in the first side image IMAGE1, and the second side image ( IMAGE2) calculates the 21st position angle ANG21 based on the position of the second reference point, the first position information P_C1 of the first camera device 311, and the second position information of the second camera device 312 The two-dimensional position information P_OBJ_XY of the input object OBJ may be generated based on the (P_C2), the eleventh position angle ANG11 and the twenty-first position angle ANG21. In addition, the method of FIG. 14 calculates the twelfth position angle ANG12 based on the position of the first reference point in the first side image IMAGE1, and based on the position of the second reference point in the second side image IMAGE2. The second position angle ANG22 is calculated, and the first position information P_C1 of the first camera device 311, the second position information P_C2 of the second camera device 312, and the twelfth position angle ANG12 ) And depth information (P_OBJ_Z) of the input object OBJ may be generated based on the 22nd position angle ANG22.

The method of FIG. 14 may determine whether spatial touch input is generated by the input object based on the 3D position of the input object, and may generate a touch event (S1150). For example, it is determined whether the depth (that is, the distance from the front of the display device to the input object) among the three-dimensional positions of the input object is within a reference range, and if it is within the reference range, the display device corresponding to the input object It may be determined that a touch input has occurred at a point (or a region). When a function to be executed (eg, an image to be displayed, etc.) to be executed in response to the touch input exists, the method of FIG. 14 may generate a touch event that executes the function.

16 is a flowchart illustrating another example of a spatial touch sensing method performed on the display device of FIG. 12.

6, 12, 13 and 16, the method of FIG. 16 may pre-process side images, respectively (S1310). Here, the side images may be obtained from the first to fourth camera devices 311 to 314 described with reference to FIG. 12.

Thereafter, the method of FIG. 16 may detect an input object from each of the pre-processed images (ie, pre-processed side images) (S1120).

The method of FIG. 16 may determine whether the input object includes a plurality of sub-objects (S1330). Here, the sub-objects may be divided or separated from each other, and each may correspond to an image that can be recognized as one input object.

When the input object includes only one sub-object, the method of FIG. 16 determines that there is only one touch input, and as described with reference to FIG. 14, calculates a three-dimensional position of the input object (S1350) , A touch event may be generated based on the 3D position (S1360).

Alternatively, when the input object includes a plurality of sub-objects, the method of FIG. 16 may determine that multi-touch input exists and determine one sub-object as an input object based on the first condition (S1340). . Here, the first condition may be a feature common to grouped camera devices, as described with reference to FIG. 13, for example, as described with reference to FIG. 13, an input object adjacent to a specific side of the display device To determine, it may be a condition for selecting an input object having the largest or smallest position angle.

Referring to FIG. 13, for example, the method of FIG. 16 selects a first input object having the smallest position angle from first and second side images (or pre-processed first and second side images), and , A second input object having the smallest position angle may be selected from the third and fourth side images (or pre-processed third and fourth side images).

Thereafter, the method of FIG. 16 may calculate a 3D position of the determined (or selected) input object (S1350). For example, a 3D position of the first input object may be calculated based on the position angles of the first input object selected from the first and second side images. Similarly, a three-dimensional position of the second input object may be calculated based on the position angles of the second input object selected from the third and fourth side images.

The method of FIG. 16 may determine whether spatial touch input is generated by the input object based on the 3D position of the input object, and may generate a touch event (S1360). For example, the method of FIG. 16 determines whether at least one of the three-dimensional position of the first input object and the three-dimensional position of the second input object satisfies the touch input condition, and generates a touch event based on whether the condition is satisfied. Can be created.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, a person skilled in the art to which the present invention pertains may be implemented in other specific forms without changing the technical spirit or essential features of the present invention. You will understand. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

1: display device
10: spatial touch sensing module
100: display unit
130: housing
131: main body
132: top cover
133: side cover
134: support
300: input detection unit
311, 312, 313, 314: first to fourth camera devices
320: infrared light source
500: control unit
700: sensory provider

Claims (9)

Display panel;
A first camera device disposed in front of the display panel and generating a first side image of a sensing space from a display surface of the display panel to a predetermined distance;
A second camera device disposed in front of the display panel and spaced apart from the first camera device to generate a second side image of the sensing space; And
And a control unit for recognizing a non-contact spatial touch input within the sensing space based on the first and second side images.
The control unit,
An extraction unit detecting a first input object located in the detection space from the first and second side images; And
And a calculator configured to calculate 3D position information of the first input object based on first position information in the first side image of the first input object and second position information in the second side image. According to claim 1, The extraction unit,
A first object image corresponding to the first input object is detected from the first side image based on a preset pattern,
A first reference point of the first object image is determined based on a predetermined feature point based on a predetermined pattern,
A second object image corresponding to the first input object is detected from the second side image based on a preset pattern,
A display device that determines a second reference point of the second object image based on a predetermined feature point based on a predetermined pattern. The method of claim 2, wherein the calculation unit,
A first position angle is calculated based on the position of the first reference point in the first side image, and a second position angle is calculated based on the position of the second reference point in the second side image, and the first camera A first position calculator that generates two-dimensional position information of the first input object based on the first position information of the device, the second position information of the second camera device, the first position angle, and the second position angle ; And
A third position angle is calculated based on the position of the first reference point in the first side image, and a fourth position angle is calculated based on the position of the second reference point in the second side image, and the first camera Calculating a second position to generate depth information of the first input object based on the first position information of the device, the second position information of the second camera device, the third position angle and the fourth position angle Including wealth,
The first position angle is an angle between one surface of the sensing space and the first input object based on the first camera device,
The second position angle is an angle between the first surface of the sensing space and the first input object based on the second camera device,
The third position angle is an angle between the display panel and the first input object based on the first camera device,
The third position angle is an angle between the display panel and the first input object based on the first camera device,
The 3D position information of the first input object includes the 2D position information and the depth information. According to claim 2, The extraction unit,
A tracking unit that tracks a change over time of the first reference point; And
And a correction unit that corrects the change of the first reference point using a Kalman filter or a low frequency filter. According to claim 1, The control unit,
A background removing unit generating a first pre-processed image by removing a background image from the first side image;
A noise removing unit that binarizes the first pre-processed image to generate a first original image, respectively; And
Selecting one of the candidate object images as a shadow image based on at least one of the size of the candidate object images included in the first original image, the symmetry between the candidate object images, and the placement positions of the candidate object images, Further comprising a shadow removing unit for removing the shadow image from the first original image to generate a first shadow removal image,
The first shadow removal image is the first side image, and a display device provided to the extraction unit. The method of claim 1, wherein the extraction unit, when the first input object includes a plurality of sub-objects that are mutually separated, a first sub-object adjacent to one side of the display panel among the sub-objects as a first input object. Display device to choose. The method of claim 6,
A third camera device disposed in front of the display panel and spaced apart from the first and second camera devices to generate a third side image of the sensing space; And
Further comprising a fourth camera device that is disposed spaced apart from the first to third camera devices in front of the display panel, and generates a fourth side image of the sensing space,
The extraction unit detects a second input object located in the sensing space from the third and fourth side images,
The calculating unit calculates 3D position information of the second input object based on the third position information in the third side image of the second input object and the fourth position information in the fourth side image. According to claim 1, The extraction unit,
A shape analysis unit configured to perform shape analysis on at least one of the first side image and the second side image when a short-distance partial image of the first input object; and
And at least one of the first side image and the second side image is a full distance image of the first input object, the display device further includes a ratio analysis unit performing ratio analysis thereon. In the spatial touch sensing method performed in the display device including the first and second camera devices are spaced apart from each other in front of the display panel,
Generating first and second side images of a sensing space from a display surface of the display panel to a predetermined distance through the first and second camera devices;
Detecting an input object located in the sensing space from the first and second side images; And
And calculating three-dimensional location information of the input object based on first location information in the first side image of the input object and second location information in the second side image.

Images