KR102158613B1 - Method of space touch detecting and display device performing the same - Google Patents

Abstract

The display device includes a display panel. The first camera device is disposed in front of the display panel, and generates a first side image of a sensing space from a display surface of the display panel to a preset distance. The second camera device is disposed spaced apart from the first camera device in front of the display panel, and generates a second side image of the sensing space. The controller recognizes a non-contact spatial touch input within the sensing space based on the first and second side images. Here, the control unit includes an extraction unit that detects a first input object located in the sensing space from the first and second side images, and the first position information in the first side image and the second side image of the first input object. And a calculator configured to calculate 3D location information of the first input object based on the second location information.

Description

A method of sensing spatial touch and a display device that performs it {METHOD OF SPACE TOUCH DETECTING AND DISPLAY DEVICE PERFORMING THE SAME}

The present invention relates to a display device, and more particularly, to a spatial touch sensing method for sensing a spatial touch and a display device for performing the same.

A touch screen installed on a display device to input data by simply touching it using a finger or a pen is widely used.

The touch screen detects the occurrence of a user input by adding a touch panel to the display panel and recognizing that the characteristics of a corresponding region (ie, a region touched by a finger, etc.) change when a finger or a pen touches it.

Since the touch screen as described above generally employs a method of operating in a two-dimensional plane in which the display screen and the touch panel are disposed on the same surface, there is a limitation in providing various expressions of the user interface.

Accordingly, a problem to be solved by the present invention is to provide a display device capable of sensing a three-dimensional touch input.

Another problem to be solved by the present invention is to provide a spatial touch sensing method capable of sensing a three-dimensional touch input.

The problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A display device according to an exemplary embodiment of the present invention for solving the above problem includes: a display panel; A first camera device disposed in front of the display panel and generating a first side image of a sensing space from a display surface of the display panel to a preset distance; A second camera device disposed in front of the display panel and spaced apart from the first camera device, and generating a second side image of the sensing space; And a controller configured to recognize a non-contact type spatial touch input within the sensing space based on the first and second side images. Here, the control unit may include an extraction unit configured to detect a first input object located in the sensing space from the first and second side images; And a calculator configured to calculate 3D location information of the first input object based on first location information in the first side image and second location information in the second side image of the first input object.

The extraction unit may detect a first object image corresponding to the first input object from the first side image based on a preset pattern, and a first object image of the first object image based on a feature point predefined in a preset pattern. 1 A reference point is determined, a second object image corresponding to the first input object is detected from the second side image based on a preset pattern, and the second object image is based on a predefined feature point in a preset pattern. The second reference point of can be determined.

The calculator calculates a first position angle based on the position of the first reference point in the first side image, and calculates a second position angle based on the position of the second reference point in the second side image, The first position information of the first camera device, the second position information of the second camera device, the first position angle, and the second position angle to generate two-dimensional position information of the first input object 1 position calculator; And calculating a third position angle based on the position of the first reference point in the first side image, calculating a fourth position angle based on the position of the second reference point in the second side image, and the first A second position for generating depth information of the first input object based on first position information of the camera device, second position information of the second camera device, the third position angle, and the fourth position angle A calculation unit, wherein the first position angle is an angle between one surface of the sensing space and the first input object with respect to the first camera device, and the second position angle is based on the second camera device. , An angle between the one surface of the sensing space and the first input object, the third position angle is an angle between the display panel and the first input object based on the first camera device, and the third The position angle is an angle between the display panel and the first input object based on the first camera device, and the 3D position information of the first input object includes the 2D position information and the depth information. I can.

The extraction unit may include a tracking unit for tracking a change in the first reference point over time; And a correction unit for correcting the change in the first reference point using a Kalman filter or a low frequency filter.

The control unit may include a background removal unit configured to generate a first preprocessed image by removing a background image from the first side image; A noise removing unit for generating first original images by binarizing the first preprocessed images based on a preset threshold; And selecting one of the candidate object images as a shadow image based on at least one of sizes of candidate object images included in the first original image, symmetry between the candidate object images, and placement positions of the candidate object images. And a shadow removal unit configured to generate a first shadow removal image by removing the shadow image from the first original image, and the first shadow removal image may be provided to the extraction unit as the first side image.

When the first input object includes a plurality of sub-objects separated from each other, the extraction unit may select a first sub-object adjacent to one side of the display panel among the sub-objects as the first input object.

The display device includes: a third camera device disposed in front of the display panel and spaced apart from the first and second camera devices, and generating a third side image of the sensing space; And a fourth camera device disposed in front of the display panel and spaced apart from the first to third camera devices, and generating a fourth side image of the sensing space. In this case, the extraction unit detects a second input object located in the sensing space from the third and fourth side images, and the calculation unit includes third position information of the second input object in the third side image and 3D location information of the second input object may be calculated based on the fourth location information in the fourth side image.

When at least one of the first side image and the second side image is a partial image near the first input object, the extracting unit performs a shape analysis, and the first side image and the When at least one of the second side images is the entire distant image of the first input object, a ratio analysis unit for performing ratio analysis may be further included.

The spatial touch sensing method according to an exemplary embodiment for solving the above other problems is performed in a display device including first and second camera devices disposed to be spaced apart from each other in front of a display panel. The method includes: generating first and second side images of a sensing space from a display surface of the display panel to a preset distance through the first and second camera devices; Detecting an input object located in the sensing space from the first and second side images; And calculating 3D location information of the input object based on first location information in the first side image and second location information in the second side image of the input object.

Details of other embodiments are included in the detailed description and drawings.

The display device according to the exemplary embodiment of the present invention obtains side images of the sensing space from the front of the display unit and calculates a three-dimensional position of the input object in the sensing space from the side images. 3D touch input can be detected.

Effects according to the embodiments of the present invention are not limited by the contents illustrated above, and more various effects are included in the present specification.

1 is a perspective view illustrating a display device according to an exemplary embodiment of the present invention.
2A and 2B are exploded perspective views illustrating an example of a display unit included in the display device of FIG. 1.
2C is a bottom side view illustrating an example of the display device of FIG. 1.
FIG. 2D is an enlarged view of area Q1 of FIG. 1.
3A to 3C are diagrams illustrating an example of a sensing space formed by an input sensing unit included in the display device of FIG. 1.
4 is a diagram illustrating a display device according to other exemplary embodiments of the present invention.
5 is a plan view illustrating an example of a sensing space formed by the display device of FIG. 4.
6 is a block diagram illustrating an example of a control unit included in the display device of FIG. 1.
7A to 7F are diagrams illustrating a process of detecting an input object by the controller of FIG. 6.
8A and 8B are schematic diagrams illustrating an input image of a pointer according to a location.
9 is a schematic diagram illustrating a comparison between an original image of a proximity pointer and a converted image.
10 is a schematic diagram showing a method of analyzing a ratio of a converted image.
11A and 11B are diagrams illustrating an example of a sense providing unit included in the display device of FIG. 1.
12 is a diagram illustrating a display device according to another exemplary embodiment of the present invention.
13 is a diagram illustrating a configuration in which the display device of FIG. 12 detects a multi-input.
14 is a flowchart illustrating an example of a spatial touch sensing method performed in the display device of FIG. 1.
15A is a flowchart illustrating an example of a process of preprocessing a side image in the method of FIG. 14.
15B is a flowchart illustrating an example of a process of detecting an input object in the method of FIG. 14.
16 is a flowchart illustrating another example of a spatial touch sensing method performed in the display device of FIG. 12.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms different from each other, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have it, and the invention is only defined by the scope of the claims.

Although the first, second, and the like are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a perspective view illustrating a display device according to an exemplary embodiment.

Referring to FIG. 1, the display device 1 includes a display unit 100, an input detection unit 300, and a control unit 500.

The display unit 100 may provide an image. Here, one surface of the display unit 100 on which an image is displayed is defined as a display surface, and the upper, lower, left, and right sides of the display device 1 are based on the display surface of the display unit 100 shown in FIG. 1 ( For example, based on a surface formed by the first direction D1 and the second direction D2), it is defined as upper, lower, left, and right. In addition, the front (or upper) of the display device 1 is defined as the direction in which the display surface is visible based on a direction perpendicular to the display surface (i.e., the third direction D3), and the rear of the display device 1 (Or, the lower part) is defined as the direction in which the display surface is not visible.

The display unit 100 may have a rectangular planar shape. The display unit 100 may be a curved display module having a specific radius of curvature. For example, the display surface of the display unit 100 may be curved with a radius of curvature of about 1.5 m. As shown in FIG. 1, the display unit 100 may have a concave shape from the front to the rear, or may have a convex shape on the contrary. However, the present invention is not limited thereto, and the display unit 100 may be a flat display module having a flat display surface. Hereinafter, a case where the display unit 100 is a curved display module will be described as an example.

The display unit 100 may be a liquid crystal display device (LCD), an organic light emitting display device (OLED), a plasma display device (PDP), a field emission display device (FED), or the like according to a method of emitting light. A more specific configuration of the display unit 100 will be described later with reference to FIG. 2A.

The input sensing unit 300 may generate sensing data for a sensing space (or sensing region, see FIG. 3A) set in front of the display unit 100. Here, the sensing space has an area equal to or similar to the display surface of the display unit 100 (based on a plane or curved surface parallel to the display surface) and within a preset distance (eg, maximum 1m) from the display surface of the display unit 100. A space in which it is located is a three-dimensional space having a predetermined volume, and may be a space in which a non-contact air touch is input by the input object OBJ. The input object OBJ may include a pointer. The pointer is an indication means indicating a direction or location, and a user's finger, a pen, an indicator rod, a baton, an antenna, a stick, etc. may be used as a pointer. In the drawings, as examples of the pointer, a user's hand (or finger) and a pen are illustrated, but it is obvious that examples of applicable pointers are not limited thereto.

The input sensing unit 300 may generate sensing data for a sensing space in which the input object OBJ is located. The sensing data may be used to generate a touch event by the controller 500 to be described later.

In embodiments, the input sensing unit 300 is disposed in front of the display unit 100 and may acquire side images (or side image data) of the sensing space.

In an embodiment, the input detection unit 300 may include a first camera device 311 and a second camera device 312. The first camera device 311 is disposed in front of the display unit 100, but is disposed toward a first gaze direction DR1 that crosses the display surface of the display unit 100, and generates a first side image of the sensing space. can do. The first gaze direction DR1 may be a direction crossing the display surface in an oblique direction (ie, a direction inclined with respect to the side of the display surface), and, for example, corners (or corners) of the display unit 100 It may be a direction toward the center of the area of the display surface from an edge of one of the parts).

Similarly, the second camera device 312 is disposed to be spaced apart from the first camera device 311 in front of the display unit 100, but toward the second line of sight direction DR2 that crosses the display surface of the display unit 100 It is disposed, and a second side image of the sensing space may be generated. Here, the second gaze direction DR2 intersects the first gaze direction DR1, and for example, the second gaze direction DR2 may be perpendicular to the first gaze direction DR1 or may form an obtuse angle. . The second line of sight direction DR2 may be a direction crossing the display surface in an oblique direction, and for example, may be a direction from the other one of the corners of the display unit 100 toward the center of the area of the display surface.

As shown in FIG. 1, the first camera device 311 may be disposed at the upper left corner of the display unit 100, and the gaze axis of the first camera device 311 may be disposed toward the center of the area of the display surface. have. Similarly, the second camera device 312 may be disposed at the upper right corner of the display unit 100, but may be disposed such that the gaze axis of the second camera device 312 faces the center of the area of the display surface.

The first and second camera devices 311 and 312 may have a view angle (or field of view) of 90 degrees or more. In this case, each of the first and second camera devices 311 and 312 may capture a sensing space in front of the display unit 100 without a shaded area.

For reference, a general camera device used to detect an input object (OBJ) based on an image (or vision) is disposed from one side of the general display device toward the front of the display device to provide a front image of the comparison sensing space. To obtain. Here, the comparative sensing space, which is a space corresponding to the sensing space according to the embodiment, is usually spaced apart from the surface of the display unit 100 by a distance equal to or greater than the minimum focal length of the camera device. That is, according to the focal length of the lens of a general camera device, since the user's hand must be separated from the general camera device by 20 cm or more to accurately detect the user's hand, the input object (OBJ) is Is difficult to detect. In order to overcome this limitation, a camera device having a short focal length lens may be used, but the camera device has a narrow angle of view, so it is difficult to detect the input object OBJ from the entire front of the display device.

On the other hand, since the input sensing unit 300 according to the exemplary embodiments is disposed to be spaced apart from each other, and detects the position by acquiring side images of the sensing space, not the front image, the space adjacent thereto ( For example, all input objects OBJ located within a space within 1m from the display surface of the display unit 100 may be detected.

Meanwhile, in FIG. 1, the first and second camera devices 311 and 312 are shown to be disposed at the upper left corner and the upper right corner of the display unit 100, but these are exemplary and are not limited thereto.

In addition, in FIG. 1, the input detection unit 300 is shown to include two camera devices 311 and 312, but this is exemplary, and the input detection unit 300 includes three or more camera devices. May be. This will be described later with reference to FIG. 12.

In one embodiment, the input sensing unit 300 may further include an infrared light source 320. The infrared light source 320 may project infrared light into the sensing space. Here, the infrared light has a wavelength of about 940 nm, and can be distinguished from the infrared light having a wavelength of 850 nm used in a surveillance camera. The infrared light source 320 may include infrared LEDs having a relatively wide projection angle or infrared laser light sources having a relatively large projection density.

When the infrared light source 320 is provided, the first camera device 311 and the second camera device 312 may be implemented as infrared cameras, or may include a visible light cut filter that blocks visible light. Accordingly, sensitivity to side images is improved, and the input object OBJ can be more easily detected by infrared light reflected by the input object OBJ. On the other hand, the display device 1 is applied to an entertainment device such as a slot machine and can be used in an entertainment venue such as a casino. The input object (OBJ) is performed only by processing a general visible image by lighting and images of high luminance in the entertainment venue. May not be easily detected. Accordingly, the input sensing unit 300 according to the embodiments uses infrared light different from visible light (also, infrared light having a wavelength different from infrared light used in a surveillance camera of an entertainment venue), and thus the input object OBJ ) Can be detected more accurately.

1, the infrared light source 320 may be disposed adjacent to the upper side of the display unit 100. However, this is exemplary, and the infrared light source 320 may be disposed adjacent to the lower side of the display unit 100, or may be disposed on the upper and lower sides of the display unit 100, respectively. In addition, the infrared light source 320 may be disposed on the left side or the right side of the display unit 100.

The control unit 500 detects the input object OBJ from side images (ie, detection data acquired by the input detection unit 300), and inputs the input object based on the position information of the input object OBJ in the side images. 3D location information of the object OBJ is calculated, and a spatial touch input or a touch event may be generated based on the 3D location information. For example, the controller 500 detects an input object by image processing side images, and position information of each of preset camera devices 300: 311 and 312, and position information of the input object in the side images (or 3D location information of the input object may be calculated based on direction information), and a touch event may be generated based on a change in the 3D location information (eg, change in the third direction D3).

Further, the controller 500 provides image data to the display unit 100 and may control or change the image data in response to a touch event. Furthermore, the controller 500 may generate a sensory control signal in response to a touch event.

In some embodiments, the display device 1 may further include a sense providing unit 700. The sense providing unit 700 may provide a specific sense (eg, tactile sense) to the input object OBJ.

The sense providing unit 700 may be disposed in front of the display unit 100. As shown in FIG. 1, the sensory providing unit 700 is disposed adjacent to the lower side of the display unit 100 and may provide a sensory signal to the sensing space. For example, the sense providing unit 700 includes ultrasonic generators (or ultrasonic transducers), and the ultrasonic generators emit ultrasonic waves from the lower side of the display unit 100 toward the upper side of the display unit 100. I can.

When the input object OBJ enters the sensory area (eg, an area affected by ultrasound, a space similar to or included in the sensing space), the end of the input object OBJ (ie, within the sensory area) is The entered part) may vibrate, and the user may recognize that a touch event is generated based on the vibration of the input object OBJ (eg, vibration of the hand).

The sensory providing unit 700 may generate ultrasonic waves based on the sensory control signal provided by the controller 500. That is, the sense providing unit 700 may generate ultrasonic waves only when a touch event occurs. Alternatively, the sensory providing unit 7000 may periodically generate ultrasonic waves irrespective of the sensory control signal.

In an embodiment, the sensory providing unit 700 may change the intensity of the ultrasound (ie, the size of a tactile sense) based on a distance between the input object OBJ and the display unit 100. For example, the sensory providing unit 700 may increase the intensity of ultrasound as the input object OBJ is adjacent to the display unit 100.

As described with reference to FIG. 1, the display device 1 acquires side images of the sensing space through the input detection unit 300, and extracts the input object OBJ from the side images through the control unit 500. And, based on the 3D position of the input object OBJ, a spatial touch input may be recognized or a touch event may be generated.

In addition, the input detection unit 300 detects the input object OBJ adjacent to the display surface of the display unit 100 by acquiring side images of the sensing space, and moves in the third direction D3 of the input object OBJ. The position change (or the distance from the display surface to the input object OBJ) can be more accurately detected.

Further, the first and second camera devices 311 and 312 included in the input detection unit 300 are spaced apart from each other and are disposed adjacent to different corners of the display unit 100, so that the display surface without a shaded area The input object (OBJ) can be detected in the entire sensing space in front of.

Meanwhile, the input sensing unit 300 and the control unit 500 (and the sense providing unit 700) may be implemented as one module (eg, a spatial touch sensing module).

2A and 2B are exploded perspective views illustrating an example of a display unit included in the display device of FIG. 1. 2C is a bottom side view illustrating an example of the display device of FIG. 1. FIG. 2D is an enlarged view of area Q1 of FIG. 1.

2A to 2D, the display unit 100 may include a display module 110 and a housing 130: 131, 132, 133, and 134.

The display module 110 is implemented as a liquid crystal display module, and may include a liquid crystal display panel 111, a light source 113, a back plate 115, and a diffusion sheet 117, as shown in FIG. 2B.

The liquid crystal display panel 111 may be a curved display panel that is concavely curved left and right with respect to a vertical axis. The liquid crystal display panel 111 may include a pixel electrode and a common electrode facing each other, a liquid crystal layer (or liquid crystal cells) disposed therebetween, transistors that transmit voltage to the pixel electrode, color filters, and polarizing members. .

The light source 113 may provide light to the liquid crystal display panel 111. The light source 113 includes a plurality of LEDs, and the LEDs may be attached to the backplate 115 in a matrix form. 2 shows that the light source 113 is disposed in a direct type, the light source 113 may be disposed in an edge type.

The back plate 115 supports the light source 113 and may cover the liquid crystal display panel 111. The back plate 115 is formed along the circumferential direction of the liquid crystal display panel 111 on a concentric circle of the same origin as the origin of the liquid crystal display panel 111 (that is, the center of a virtual circle obtained by extending the curved surface). It can have a set width. The back plate 115 may include a plurality of rectangular plates, and each of the plates may be disposed to face the origin of the liquid crystal display panel 111.

The optical sheet 117 is disposed between the liquid crystal display panel 111 and the light source 113. The optical sheet 117 may include a diffusion sheet, a prism sheet, and a brightness enhancement sheet.

Meanwhile, although not shown, the display module 110 may further include a window disposed in front of the liquid crystal display panel 111 to protect the liquid crystal display panel 111.

Referring back to FIG. 2A, the housing 130 may accommodate the display module 110 and protect the display module 110 from the rear and side surfaces of the display module 110. In addition, the housing 130 accommodates the input sensing unit 300 and/or the sense providing unit 700, or the input sensing unit 300 and/or the sense providing unit 700 is mounted in the housing 130. ) Can be.

The housing 130 may include a body part 131, an upper cover 132, a side cover 133, and a support part 134.

The main body 131 may have a rectangular planar shape corresponding to the display module 110, may be bent together in a direction in which the display device 1 is bent, and may include a storage space formed by recessing the front surface. The display module 110 may be accommodated in the storage space.

At least a portion of the rear surface of the main body 131 may be cut off or open, and a module-type control unit 150 may be disposed at the cut or open portion of the main body 131.

The upper cover 132 is disposed on the upper side (or upper front surface) of the main body 131 and may include an empty space formed therein between the main body 131 and the main body 131. The input detection unit 300 may be disposed in the empty space. The upper cover 132 and the input sensing unit 300 may be configured as one module, and may be disposed or coupled to the upper side of the main body 131.

As shown in FIG. 2C, an opening OP communicating the empty space and the outside may be formed between the upper cover 132 and the main body 131 at the lower side of the upper cover 132. The housing 130 may further include a transparent plate 132-1 covering at least a portion of the opening. The upper cover 132 is made of an opaque and rigid material, so that the input sensing unit 300 can be prevented from being exposed to the outside. Infrared light emitted from the input sensing unit 300 (or the infrared light source 320) may be transmitted through the transparent plate 132-1 disposed in the opening OP to be provided to the sensing space.

On the other hand, part of the opening OP of the upper cover 132 is not covered by the transparent plate 132-1, and the first and second camera devices 311 and 312 (for example, camera devices ( 311, 312) each lens) may be exposed to the outside. With such a structure, the infrared light reflected by the input object OBJ may be prevented from being totally reflected by the transparent plate 132-1 (or the refractive index of the transparent plate 132-1 ).

Referring back to FIG. 2A, the side cover 133 may have a rectangular planar shape and may be disposed on the side of the body portion 131. The side cover 133 may block light (eg, infrared light) flowing from the side of the display module 110. In addition, the side cover 133 may provide an installation area of the input sensing unit 300 (or the first and second camera devices 321 and 322 ).

In one embodiment, the side cover 133 may define a sensing space. For example, in side images acquired from the first and second camera devices 321 and 322, it may be determined that only the input object OBJ displayed by overlapping with the side cover 133 exists in the sensing area. have.

The support 134 (or the support member) may support the first and second camera devices 311 and 312 and the infrared light source 320. The support part 134 may include a first support part 134-1 for supporting the first and second camera devices 311 and 312 and a second support part (not shown) for supporting the infrared light source 320. .

As shown in FIG. 2D, the first support part 134-1 is from the fixing part 134a fixed to the upper surface of the main body 131 (or the display module 110) and the fixing part 134a. A first extension portion 134b extending along an oblique direction (ie, a fourth direction D4, a direction perpendicular to the line of sight axis of the first camera device 311, or a direction forming an acute angle with the fixing portion 134a) ), and a second extension part 134c extending from the first extension part 134b to the front of the display module 110. The first camera device 311 (or the second camera device 312) may be fixed to the inner surface of the second extension part 134c.

The horizontal angle of the first camera device 311 (that is, the horizontal component of the gaze axis of the first camera device 311) is adjusted according to the alignment of the fixing part 134a and the main body part 131, and the fixing part 134a The vertical angle of the first camera device 311 (ie, a vertical component of the line of sight of the first camera device 311) may be adjusted according to the bent angle of the first extension part 134b based on.

Meanwhile, in FIG. 2D, the first support 134-1 is shown to be disposed on the upper side of the main body 131, but is not limited thereto, and the first support 134-1 is the main body 131 It may be disposed on the side or side cover 133 of the.

The second support part (not shown) may extend upward from the upper side of the main body 131 (or the display module 110 ), and may be formed by bending toward the front of the display module 110. The second support part has a "L" shape (or inverse "L" shape), and an infrared light source on the inner surface of the second support part (not shown) (that is, a surface facing the upper surface of the main body 131) 320 may be placed. The second support part prevents the infrared light emitted from the infrared light source 320 from traveling in the horizontal direction (that is, the infrared light directly entering the first and second camera devices 311 and 312). It may also include a side light blocking member.

In some embodiments, the display unit 100 may further include a three-dimensional filter unit (not shown). The three-dimensional filter unit may be disposed in front of the display module 110. For example, the three-dimensional filter unit may have a radius of curvature that is substantially the same as or similar to that of the display module 110 and may be disposed in front of the display module 110. The three-dimensional filter unit may be disposed in close contact with the front surface of the display module 110 or spaced apart from the front surface of the display module 110.

The three-dimensional filter unit (not shown) may include a lenticular lens sheet. By providing the left eye image of the display module 110 to the user's left eye and the right eye image to the user's right eye by the lenticular lens, a 3D stereoscopic image may be provided to the user. When the lenticular lens is formed to be concave with the same curvature as that of the display module 110, an optical illusion phenomenon such as the existence of an image in the concave space may occur. Accordingly, the image appears to be displayed on the space due to the optical illusion, and the sense of space can be maximized by the user touching the image by putting his hand in a certain depth of the concave space.

As described with reference to FIGS. 2A to 2D, the display unit 100 covers the input sensing unit 300 using the upper cover 132, but at least a part of the upper cover 132 is the input sensing unit 300. The first and second camera devices 311 and 312 of are exposed so that side images reflecting all of the light reflected by the input object OBJ can be generated. In addition, the display unit 100 includes side covers 133a and 133b disposed on the side of the display module 110 to block light from outside, and to more easily define or form a sensing space. I can.

3A to 3C are diagrams illustrating an example of a sensing space formed by an input sensing unit included in the display device of FIG. 1. 3B and 3C illustrate a plane of the sensing space.

First, referring to FIGS. 3A and 3B, the first and second camera devices 311 and 312 are disposed in front of the display unit 110 as described above, but the upper left and upper right corners of the display unit 110 Each can be placed in the corner. In addition, the first and second camera devices 311 and 312 may be disposed such that the gaze axes of the first and second camera devices 311 and 312 face the center of the area of the display surface. The first and second camera devices 311 and 312 may have a view angle of 90 degrees or more, for example, 90 degrees or 140 degrees.

The sensing space SA may be formed in a region where the photographing region of the first camera device 311 and the photographing region of the second camera device 312 overlap. As the sensing space SA is adjacent to the first and second camera devices 311 and 312, the width (or thickness, that is, the length in the direction perpendicular to the display surface of the display unit 110) is smaller, The width may increase as the distance from the first and second camera devices 311 and 312 increases. However, the present invention is not limited thereto, and the shape and width of the sensing space may vary according to the curvature of the display unit 110 and the arrangement direction of the first and second camera devices 311 and 312.

As illustrated in FIG. 3B, the sensing space SA may have a planar shape of an arc (or sector) based on a point (refer to “P0”) positioned in front of the display device 1.

In one embodiment, the edge of the viewing angle of the first camera device 311 and the edge of the viewing angle of the second camera device 312 are the origin of the display device 1 (or the display surface of the display unit 100) (P0 ) Or the origin P0 of the display device 1 may be located within the viewing range of the first camera device 311 and the viewing range of the second camera device 312. In this case, the sensing space SA has the same thickness (i.e., the length in a direction perpendicular to the display surface) over the entire front surface of the display device 1, and is located in the front surface of the display device 1 according to the shape of the sensing space. An input range of spatial touch having the same maximum depth may be provided. For example, when the display device 1 has a display surface of 55 inches and a radius of curvature of about 1.5 m, the first and second camera devices 311 and 312 are located at the center of the area of the display surface (' P1'), the edge of the viewing angle of the first and second camera devices 311 and 312 may pass through the origin P0 of the display device 1. In this case, the sensing space SA has an arc shape with a radius of about 1.5 m, and the display device 1 is based on the user (or the user's gaze) located at the origin P0 of the display device 1 As a result, an input range of spatial touch having the same depth DE1 of at most 1.5m over the entire front area of the display device 1 may be provided.

Meanwhile, when the infrared light source 320 projects infrared light to an area having a certain width (eg, 30 cm) in front of the display device 1, the display device 1 A sensing space SA having the same depth DE2 (eg, a maximum depth of 30 cm) may be formed in the entire area.

As described with reference to FIGS. 3A to 3C, when the viewing angles of the first and second camera devices 311 and 312 are disposed to pass through the origin P0 of the radius of curvature of the display device 1, a separate Without image processing, the sensing space SA having the same depth ('DE2' in FIG. 3C) is formed over the entire front surface of the display device 1, or the same maximum depth (FIG. 3B) over the entire front surface of the display device 1 An input range of spatial touch having'DE1') of may be provided.

4 is a diagram illustrating a display device according to other exemplary embodiments of the present invention. 5 is a plan view illustrating an example of a sensing space formed by the display device of FIG. 4.

1, 4 and 5, the display device 1_6 is different from the display device 1 of FIG. 1 in that it includes a flat display unit 100. Except for the flat display unit 100 (and configurations corresponding to the flat display unit 100), the display device 1_6 of FIG. 4 is substantially the same as or similar to the display device 1 of FIG. Decided not to repeat.

The display device 1_6 includes first and second camera devices 311 and 312, and the first camera device 311 is disposed adjacent to the upper left corner of the display unit 100, and the second camera device ( 312 may be disposed adjacent to the upper right corner of the display unit 100.

However, the first camera device 311 is disposed adjacent to the end of the first side cover 133a (or left side cover) coupled to the left side of the display unit 100, and the second camera device 312 is It may be disposed adjacent to the end of the second side cover 133b (or right side cover) coupled to the right side of the display unit 100. That is, the first and second camera devices 311 and 312 have the width of the first and second side covers 133a and 133b from the display unit 100 (or display surface) (that is, perpendicular to the display surface). It can be arranged spaced apart by the length in the direction).

As the display unit 100 is implemented in a flat type rather than a curved type, when the first and second camera devices 311 and 312 are disposed adjacent to the display surface, based on FIG. 5B, at both sides of the display unit 100 Since the thickness of the sensing space is relatively small, a spatial touch input having a sufficient depth may not be provided on both sides of the display unit 100. On the contrary, when the first and second camera devices 311 and 312 are arranged facing the front, the sensing space SA becomes unnecessarily large and unnecessary information (eg, objects separated by 1 m or more from the display surface) As this is sensed, the load of the display device may increase.

Therefore, in consideration of the depth of the spatial touch input to be provided on both sides of the display unit 100, the first and second camera devices 311 and 312 are spaced forward from the display unit 100, 2 It may be disposed on the side covers 133a and 133b, respectively. For example, as the depth of the spatial touch input to be provided increases, the first and second camera devices 311 and 312 may be disposed more spaced apart from the display unit 100, and the first and second side surfaces The width of the covers 133a and 133b (that is, the length in a direction perpendicular to the display surface) may also be increased.

On the other hand, when the first and second camera devices 311 and 312 are separated from the display unit 100, shadow areas may occur on both sides of the display unit 100. Accordingly, the first and second camera devices 311 and 312 may be arranged to face a reference point located at the rear of the display surface rather than the area center of the display surface.

6 is a block diagram illustrating an example of a control unit included in the display device of FIG. 1. 7A to 7F are diagrams illustrating a process of detecting an input object by the controller of FIG. 6.

7A illustrates an example of a user's hand positioned in the sensing space of the display device 1 as an example of a pointer as an input object, and FIG. 7B shows a first camera device included in the display device 1 of FIG. 7A A first side image acquired at 311 is shown, and FIG. 7C shows a second side image acquired by a second camera device 312 included in the display device 1 of FIG. 7A. Infrared light (that is, the infrared light projected from the infrared light source 320 described with reference to FIG. 1 to the sensing space) is reflected or scattered by the user's hand located in the sensing space, so that the user's hand may appear relatively bright. have.

Referring to FIG. 6, the control unit 500 may include a preprocessing unit 610, an extraction unit 620, and a calculation unit 630. In addition, the control unit 500 may further include an event generating unit 640. Since the image processing processes for the first and second side images IMGAE1 and IMAGE2 are substantially the same or similar to each other, the image processing processes for the first and second side images IMGAE1 and IMAGE2 are mutually common. The process will be described based on the first side image IMAGE1.

The preprocessor 610 may remove the background image from the first side image IMAGE1, remove noise, and also remove the shadow of the input object. That is, the preprocessor 610 may pre-process the first side image so that the hand image can be easily extracted from the first side image IMAGE1.

The preprocessor 610 may include a background removing unit 611, a noise removing unit 612, and a shadow removing unit 613.

The background removal unit 611 may generate or obtain a first preprocessed image IMAGE_B1 by performing a difference calculation on a preset background image IMAGE_BACK from the first side image IMAGE1. Here, the preset background image IMAGE_BACK may be an image of an area in which only the display surface of the display unit 100 is displayed. The background removal unit 611 may convert the pixel value of the corresponding region to 0 (eg, black).

The noise removal unit 612 samples (or binarizes) the first preprocessed image IMAGE_B1 and smoothes it to obtain a first original image IMAEG_N1 from which noise has been removed. For example, the noise removal unit 612 may replace pixel values in the first preprocessed image IMAGE_B1 with 0 (eg, black) or 255 (eg, white) based on the threshold value. .

The shadow removal unit 613 may remove the shadow image by applying a shadow removal algorithm to the first original image IMAGE_N1 and generate the shadow removal image IMAGE_S1. Here, the shadow image may be an image in which a hand is reflected and displayed by the display surface of the display unit 100.

As shown in FIG. 7D, the first original image IMAGE_N1 may include candidate object images OBJ_1 and OBJ_1S. Here, the candidate object images OBJ_1 and OBJ_1S are images that can be recognized as the input object OBJ, and may have a size equal to or larger than a reference size (or a reference area). In this case, the shadow removal unit 613 determines the shadow image OBJ_1S among the candidate object images OBJ_1 and OBJ_1S based on the size, symmetry, and placement position of the candidate object images OBJ_1 and OBJ_1S, and 1 The shadow image OBJ_1S may be removed from the original image IMAGE_N1. For example, when candidate object images OBJ_1 and OBJ_1S having the same or similar shape to each other exist, a shadow image OBJ_1S having a relatively small size and adjacent to the display unit 100 may be selected and removed.

The extraction unit 620 may extract a hand image from the shadow removal image IMAGE_S1 finally output from the preprocessor 610. The extraction unit 620 may include a detection unit 621, a tracking unit 622 and a correction unit 623.

The detector 621 may detect a pointer image by matching preset patterns (eg, features of an input object or object patterns including feature points) with respect to the shadow removal image IMAGE_S1. For example, the detection unit 621 extracts a candidate object image OBJ1 having a preset size (i.e., pixels greater than or equal to a preset number) and matches the candidate object image OBJ1 with a preset pattern to obtain a hand image. Can be detected. Here, the preset patterns are, for example, a pattern corresponding to a hand with only one finger open (a pattern including feature points up to a finger, palm, and wrist), a pattern corresponding to a hand with a fist, and A pattern corresponding to the hand may be included.

In addition, the detection unit 621 of the extraction unit 620, when the image of the pointer as an input object (a hand image in the drawing) is detected, a corresponding pattern (ie, a pattern used to detect a hand image, or a corresponding pattern) A reference point of the input object image may be determined based on the feature point defined in [0104]. For example, as illustrated in FIG. 7D, when a hand image OBJ_1 corresponding to a pattern indicating a state in which one finger is opened is detected, a point at the end of the finger may be determined as the reference point P_OBJ1. For another example, when a hand image corresponding to a pattern indicating a state in which all fingers are opened is detected, the center point of the palm may be determined as the reference point P_OBJ1.

Meanwhile, the shape of the input image of the pointer such as a finger or a pen may vary according to the distance between the first and second camera devices 311 and 312. Even when the shape of the input image is changed as described above, in order to accurately recognize and detect the pointer, the detection unit 621 may further perform shape analysis, ratio (size) analysis, and/or multi-camera weight analysis. To this end, the detection unit 621 may further include a shape analysis unit, a ratio (size) analysis unit, and/or a multi-camera weight analysis unit. 8 to 10 are referred to for a detailed description thereof.

8A and 8B are schematic diagrams illustrating an input image of a pointer according to a location.

Referring to FIGS. 8A and 8B, the image type of the pointer input to the camera device 310 varies according to the distance the pointer is away from the camera device 310. That is, in the case of the user's hand, as shown in FIG. 8A, the shape of the entire hand may be input as an image image at a distance, but at a close distance, the shape of the hand may be recognized as a single mass as a partial shape. In addition, in the case of a pen, as shown in FIG. 8B, an input image for the entire shape of the pen can be obtained at a distance, but at a close distance, the shape of the pen is only partially input and thus can be viewed as a single block.

In this way, even when the pointer is located in a short distance, it is possible to identify where the pointer points on the screen by performing shape analysis. 9 is a schematic diagram illustrating a comparison between an original image of a proximity pointer and a converted image. As shown in FIG. 9, when feature points are extracted and connected to an original image in a near distance, the end of the hand or the pen has a relatively small curvature compared to the surroundings. By recognizing such a curvature, it is possible to determine where the pointer is pointing in a partial input image at a short distance. In the near-field input image, the end of the pointer may be determined as a point having the same curvature compared to a curvature of a previously stored image pattern, or may be determined as a point having the minimum curvature in a curve formed by the feature points.

10 is a schematic diagram showing a method of analyzing a ratio of a converted image. Referring to FIG. 10, ratio analysis may be performed to distinguish a pointer from a noise image even in the case of a distant image image. The ratio analysis may be performed by comparing the ratio of the vertical width (up and down width) and the horizontal width (left and right width) of the image. For example, in the case of a hand or a pen, the width in the horizontal direction is larger than the width in the vertical direction, whereas in the case of a human body, the width in the vertical direction may be analyzed as being larger than the width in the horizontal direction. As another example, the ratio analysis may be performed by comparing the ratio of the first width in the longest direction and the second width in a direction intersecting (usually perpendicular to) the first width. In the illustrated example, the hand or pen has a ratio of the first width and the second width of 2:1 or more, but the human body may be classified as having a ratio of the first width and the second width of less than 2:1. In this way, by comparing the ratio of the width in the vertical direction and the width in the horizontal direction or the ratio of the first width and the second width, the pointer can be accurately recognized and the noise image can be removed.

Meanwhile, an input image inputted to each of the camera devices 311 and 312 by a single pointer is different according to the distance. For accurate object recognition, different weights may be given for each distance to the camera devices 311 and 312. As described above, even a partial input image at a short distance may determine the end of the pointer and a direction in which the pointer is directed through shape analysis, but the accuracy may be lower than that of the entire input image at a distance. In order to compensate for this, the accuracy of object recognition may be improved by adding weight to a far input image among a plurality of input images for one pointer. For example, when the input object OBJ is located relatively closer to the first camera device 311 than the second camera device 312 in the display device 1 illustrated in FIG. 1, the second camera device 312 Object recognition can be performed by giving more weight to the image image input through ). The weight can be adjusted from a ratio of 100:0 to a ratio of 50:50 depending on the distance. The distance away from the camera devices 311 and 312 may be calculated by measuring the size of an image of the pointer, or may be calculated by receiving feedback from the three-dimensional position of the pointer calculated by the calculation unit 630 described later.

Referring back to FIGS. 6 and 7, the tracking unit 622 may track a pointer image over time using a pointer (hand in the drawing) tracking algorithm as a target object. For example, the tracking unit 622 may predict the next position of the hand image based on the movement path of the hand image (or reference point) over time, and the hand image may be calculated based on the next position of the predicted hand image. Even if the hand image is not detected quickly or the hand image is temporarily not detected, the hand image (or reference point) can be maintained for a specific time. Through this method, it is possible to prevent the hand image from being temporarily recognized and thus not performing a touch input.

The correction unit 623 may remove noise over time due to a change in the position of the reference point P_OBJ1. For example, when the tracking pointer moves quickly or the recognized reference point P_OBJ1 shakes, the correction unit 623 uses a filter such as a Kalman filter or a low pass filter to determine the location of the reference point P_OBJ1. Can be corrected.

That is, the extraction unit 620 may detect the pointer image (and/or the reference point of the pointer image) from the preprocessed side images, but may track and correct the pointer image suitable for use as a spatial touch input.

The calculator 630 may calculate a 3D position of the pointer in the sensing space based on the pointer image (or the reference point P_OBJ1). The calculation unit 630 may include a first position calculation unit 631 and a second position calculation unit 632.

The first position calculator 631 may calculate a two-dimensional position (or coordinates) of the pointer on a plane corresponding to the display surface of the display unit 100. As shown in FIG. 7A, the first position calculator 631 may calculate the eleventh position angle ANG11 of the hand image OBJ_1 based on the reference point of the hand image OBJ_1. Here, the eleventh position angle ANG11 may be an angle between the input object OBJ from the upper surface of the sensing space with respect to the first camera device 131. The first position calculator 631 may calculate the eleventh position angle ANG11 based on a distance (or the number of pixels) between the reference point of the hand image OBJ_1 from the upper side of the first side image. Similarly, as shown in FIG. 7C, the first position calculator 631 may calculate the 21st position angle ANG21 of the hand image OBJ_2 based on the reference point of the hand image OBJ_2.

Thereafter, as shown in FIG. 7E, the first position calculating unit 631 includes the first camera position information P_C1 of the first camera device 311 and the second camera position of the second camera device 312. Based on the information P_C2, the eleventh position angle ANG11, and the 21st position angle ANG21, the two-dimensional position P_OBJ_XY of the hand as a pointer may be calculated. The two-dimensional position (P_OBJ_XY) of the hand may be a point (or coordinates) where the hand is positioned on the display surface of the display unit 100.

The second position calculator 632 may calculate a depth of the pointer (or a depth value, a distance a hand is separated from the display surface) based on the display surface of the display unit 100. 7A to 7C, the second position calculator 632 calculates the twelfth position angle ANG12 and the 22nd position angle ANG22 based on the reference points of the hand images OBJ_1 and OBJ_2. I can. Here, the twelfth position angle ANG12 may be an angle between the input object OBJ from the front of the sensing space (or the display surface of the display unit 100) with respect to the first camera device 131.

Thereafter, as shown in FIG. 7F, the second position calculating unit 632 includes preset first camera position information (P_C1) of the first camera device 311 and a second camera of the second camera device 312. Based on the location information P_C2, the twelfth position angle ANG12, and the 22nd position angle ANG22, the depth of the hand, which is a pointer, P_OBJ_Z may be calculated.

Referring back to FIG. 6, the second position calculating unit 632 is based on the two-dimensional position of the hand (P_OBJ_XY) and the depth of the hand (P_OBJ_Z) calculated by the first position calculating unit 631. (P_XYZ) can be created.

Meanwhile, the event generator 640 may generate a touch event based on the three-dimensional position P_XYZ of the pointer (hand) generated by the calculator 630.

As described with reference to FIGS. 6 to 7F, the controller 600 detects a pointer (or hand) as an input object from side images through image processing, calculates a three-dimensional position of the pointer, and calculates the three-dimensional position of the pointer. A touch event may be generated based on the location.

In addition, the control unit 600 may more accurately detect the pointer by removing the shadow reflected from the display surface of the display unit 100.

Furthermore, the controller 600 may generate a more accurate touch event by tracking the pointer image and correcting the pointer image over time.

11A and 11B are diagrams illustrating an example of a sense providing unit included in the display device of FIG. 1.

Referring to FIGS. 1 and 11A, the sensory providing unit 700 may include a plurality of ultrasonic transducers 710 arranged in the front and lower sides of the display unit 100 and arranged along the first direction D1. have. The ultrasonic transducer 710 may generate directional ultrasonic waves (eg, ultrasonic waves having a frequency of 40 kHz) in the second direction D2 to apply pressure to at least a portion of the input object OBJ. That is, the sensory providing unit 700 may provide a tactile sensation to the input object OBJ using the law of acoustic radiation force.

In an embodiment, the sensory providing unit 700 may include a first ultrasonic transducer 711 and a second ultrasonic transducer 712 having different frequencies or different intensities. The first ultrasonic transducer 711 and the second ultrasonic transducer 712 may be arranged as a plurality of rows or columns. For example, the first ultrasonic transducer 711 and the second ultrasonic transducer 712 each form a row and are arranged along the first direction D1, the second ultrasonic transducer 712 is the first The ultrasonic transducer 712 may be disposed adjacent to the display unit 100.

The second ultrasonic transducer 712 may have a lower frequency or higher intensity than the first ultrasonic transducer 711. In this case, the closer the input object OBJ is to the display unit 100, the greater pressure (or tactile sensation) may be provided to the input object OBJ.

In FIG. 11A, the ultrasonic transducer 710 is shown to emit ultrasonic waves in the second direction D2, but the present invention is not limited thereto.

Referring to FIG. 11B, the sense providing unit 700_1 may include third ultrasonic transducers 713 and fourth ultrasonic transducers 714. The third ultrasonic transducers 713 and the fourth ultrasonic transducers 714 may generate and output ultrasonic waves having the same frequency (eg, 40 KHz).

The third ultrasonic transducers 713 are disposed adjacent to the left side from the front lower side of the display unit 100, and are disposed to face the fifth direction D5, so that they can emit ultrasonic waves toward the fifth direction D5. I can. Similarly, the fourth ultrasonic transducers 714 are disposed adjacent to the right side from the front lower side of the display unit 100, and are disposed to face the sixth direction D6, so that the fourth ultrasonic transducers 714 are disposed toward the sixth direction D6. Can radiate. As the sixth direction D6 crosses the fifth direction D5, the ultrasonic waves output from the third ultrasonic transducers 713 are ultrasonic waves output from the fourth ultrasonic transducers 714, and a plurality of points. Can cross in the field.

In this case, the control unit 500 determines a specific point (that is, a point to which tactile sensation should be provided) based on the three-dimensional position of the input object OBJ, and the third and fourth ultrasonic transducers based on the determined point. Some of the 713 and 714 may be operated, or a sensory control signal for operating them may be generated. For example, in order to provide a tactile sensation to the input object OBJ shown in FIG. 11B, the controller 700 may operate the first sub ultrasonic transducer 713a and the second sub ultrasonic transducer 714a. . In this case, a pressure sufficient to be recognized as a tactile sense to a user may be provided only at a point (or region) where the ultrasonic waves are overlapped. That is, the user can receive a tactile sensation only at the point where the ultrasonic waves overlap.

Meanwhile, in FIG. 11B, the third ultrasonic transducers 713 and the fourth ultrasonic transducers 714 are shown to be arranged to be separated from each other on the lower side of the display unit 100, but this is only an example and is limited thereto. no. For example, the third and fourth ultrasonic transducers 713 and 714 may be alternately arranged with each other. For another example, the third ultrasonic transducers 713 are disposed under the display unit 100, and the fourth ultrasonic transducers 714 are disposed on the left (or right) of the display unit 100, They may emit ultrasonic waves in directions that intersect each other. That is, if the third and fourth ultrasonic transducers 713 and 714 emit ultrasonic waves crossing each other at a plurality of points, their positions are not limited.

In addition, the third and fourth ultrasonic transducers 713 and 714 may be arranged along a plurality of columns (or a plurality of rows), similar to the sensory providing unit 700 illustrated in FIG. 11A.

12 is a diagram illustrating a display device according to another exemplary embodiment of the present invention. 13 is a diagram illustrating a configuration in which the display device of FIG. 12 detects a multi-input. 13 illustrates positions of camera devices 311 to 314 and input objects OBJ1 and OBJ2 on the front side of the display device.

Referring to FIG. 12, the display device 1_7 is different from the display device 1 of FIG. 1 in that it further includes third and fourth camera devices 313 and 314.

The third camera device 313 is disposed to be spaced apart from the first and second camera devices 311 and 312 in front of the display unit 100, and a third line of sight direction DR3 crossing the display surface of the display unit 100 ), it is possible to generate a third side image of the sensing space. Similarly, the fourth camera device 314 is disposed to be spaced apart from the third camera device 313 (and the first and second camera devices 311 and 312) in front of the display unit 100, but the display device ( It is disposed toward the fourth gaze direction DR4 that crosses the display surface of 1) to generate a fourth side image of the sensing space.

The third and fourth camera devices 313 and 314 may be provided in a lower cover that is substantially the same as or similar to the upper cover 132 described with reference to FIG. 2A. Further, the third and fourth camera devices 313 and 314 may be fixed by the support 134 described with reference to FIG. 2D.

The display device 1_7 may detect two input objects (eg, two hands of a user) through the first to fourth camera devices 311, 312, 313, and 314.

When the first and second input objects OBJ1 and OBJ2 are located in the sensing space in front of the display device 1_7, one of the first and second camera devices 311 and 312 is An image of one of the input objects OBJ1 and OBJ2 may not be acquired.

That is, as shown in FIG. 12, when the first input object OBJ1 and the second input object OBJ2 are located along the first gaze direction DRR1 (or the third gaze direction DRR3), Based on the first camera device 311, the second input object OBJ2 is covered by the first input object OBJ1, so the first camera device 311 generates an image for the second input object OBJ2. It may not be possible. As the first input object OBJ1 has a volume rather than a point, as shown in FIG. 13, a shadow area SZ for the first camera device 311 may occur.

Accordingly, the display device 1_7 further includes third and fourth camera devices 313 and 314, and not only the first input object OBJ1 using the third and fourth camera devices 313 and 314 In addition, the second input object OBJ2 located in the shaded area SZ may be detected.

6 and 13, position information of a first reference point P_OBJ1 for a first input object OBJ from a first side image obtained from the first camera device 311 (for example, the 11th position Only angle (ANG11)) can be obtained. Similarly, from the fourth side image acquired from the fourth camera device 314, only the positional information of the second reference point P_OBJ2 for the second input object OBJ2 (for example, the 42nd position angle ANG42) Can be obtained. Meanwhile, location information for each of the first and second input objects OBJ1 and OBJ2 may be obtained from the second and third side images acquired from the second and third camera devices 312 and 313. .

In this case, the controller 600 determines a camera device from which only location information on one input object is obtained and a camera device from which location information on two input objects is obtained as one group, and Based on the three-dimensional position of the one input object can be calculated. In addition, the remaining two camera devices may be determined as another group, and a 3D position of the other of the two input objects may be calculated based on information of the corresponding group.

For example, the second camera device 312 is grouped together with the first camera device 311 into a first group, and the first and second camera devices 311 and 312 are the first camera devices 312 of the display device 1_7. As it is positioned on the long side LS1, a three-dimensional position of the first input object OBJ1 adjacent to the first long side LS1 among the first and second input objects may be calculated. Based on the second camera device 312, since the 21st position angle ANG21 of the first input object OBJ1 is smaller than the 22nd position angle ANG22 of the second input object OBJ2, the controller 500 Is determined by determining the 21st position angle ANG21 as the position angle of the first input object OBJ1, and based on the eleventh position angle ANG11 and the 21st position angle ANG21, 3 of the first input object OBJ1 You can calculate the dimensional position.

Similarly, the third camera device 313 is grouped together with the fourth camera device 314 into a second group, and the third and fourth camera devices 313 and 314 are the second long side of the display device 1_7. According to the position at LS2, a 3D position of the second input object OBJ2 adjacent to the second long side LS2 among the first and second input objects may be calculated. Based on the third camera device 313, since the 32nd position angle ANG32 of the second input object OBJ2 is smaller than the 31st position angle ANG31 of the first input object OBJ1, the controller 500 Is determined as the 32nd position angle ANG32 as the position angle of the second input object OBJ2, and based on the 32nd position angle ANG32 and the 42nd position angle ANG42, 3 of the second input object OBJ2 You can calculate the dimensional position.

For another example, the third camera device 313 is grouped together with the first camera device 311 into a first group, and the first and third camera devices 311 and 313 are As adjacent to the first short side SS1, a 3D position of the first input object OBJ1 adjacent to the first short side SS1 among the first and second input objects may be calculated. Based on the third camera device 313, since the 31st position angle ANG31 of the first input object OBJ1 is larger than the 32nd position angle ANG32 of the second input object OBJ2, the controller 500 Determines the 31st position angle ANG31 as the position angle of the first input object OBJ1, and based on the eleventh position angle ANG11 and the 31st position angle ANG31, 3 of the first input object OBJ1 You can calculate the dimensional position.

Similarly, the second camera device 312 is grouped together with the fourth camera device 314 into a second group, and the second and fourth camera devices 312 and 314 are the second short sides of the display device 1_7. As it is positioned at SS2, a 3D position of the second input object OBJ2 adjacent to the second short side SS2 among the first and second input objects may be calculated. Based on the second camera device 312, since the 22nd position angle ANG32 of the second input object OBJ2 is larger than the 21st position angle ANG31 of the first input object OBJ1, the controller 500 Is determined by determining the 22nd position angle ANG22 as the position angle of the second input object OBJ2, and based on the 22nd position angle ANG22 and the 42nd position angle ANG42, 3 of the second input object OBJ2 You can calculate the dimensional position.

That is, the control unit 500 groups the first to fourth camera devices 311, 312, 313, and 314 into first and second groups, and the first input object among side images corresponding to the first group (Or, only information on the input object) is obtained to calculate the 3D position of the first input object, and only the second input object is obtained from the side images corresponding to the second group to calculate the 3D position of the second input object can do.

As described with reference to FIGS. 12 and 13, the display device 1_7 may set two camera devices as a group and detect one input object for each group. Accordingly, the display device 1_7 Can detect a plurality of input objects.

Meanwhile, in FIGS. 12 and 13, the display device 1_7 is shown to detect two input objects using four camera devices, but this is only an example and is not limited thereto. The display device 1_7 may accurately detect N input objects using 2N (where, N is an integer greater than or equal to 2) camera devices arranged to be spaced apart from each other.

14 is a flowchart illustrating an example of a spatial touch sensing method performed in the display device of FIG. 1. 15A is a flowchart illustrating an example of a process of preprocessing a side image in the method of FIG. 14. 15B is a flowchart illustrating an example of a process of detecting an input object in the method of FIG. 14.

1, 6, 14, and 15, the method of FIG. 14 uses the input sensing unit 300 (or the first and second camera devices 311 and 312). Side images may be acquired (S1110).

Thereafter, the method of FIG. 14 may pre-process side images (S1120). In the method of FIG. 14, as described with reference to FIG. 6, a preset background image is removed from side images (S1210), and noise is removed by binarizing and smoothing the side images (S1220), and included in the side images. The shadow image may be removed based on the size and symmetry of the candidate object images (S1230).

Thereafter, the method of FIG. 14 may detect an input object from preprocessed images (ie, preprocessed side images) (S1130). As described with reference to FIG. 6, the method of FIG. 14 detects an object image for an input object through mapping with a preset pattern, and determines a reference point of the input object based on a preset pattern from the detected object image. Then, the position (eg, position angles) of the reference point in the pre-processed images may be calculated (S1240).

With reference to FIGS. 7A to 7C, for example, the method of FIG. 14 detects a first object image OBJ_1 corresponding to the input object OBJ from the first side image IMAGE1 based on a preset pattern, and , A first reference point of the first object image OBJ_1 is determined based on a predefined feature point in a preset pattern, and a first reference point corresponding to the input object OBJ from the second side image IMAGE2 is determined based on the preset pattern. 2 The object image OBJ_2 may be detected, and a second reference point of the second object image OBJ_2 may be determined based on a feature point predefined in a preset pattern.

In addition, the method of FIG. 14 continuously tracks the position of the input object (or reference point) in the side images over time (S1250), and temporarily changes the position of the input object (or reference point) over time. It can be corrected (S1260).

Thereafter, the method of FIG. 14 may calculate a 3D position of the input object based on the object image (or reference point) (S1140). As described with reference to FIG. 6, the method of FIG. 14 includes preset location information of the first and second camera devices 311 and 312 and a reference point of the calculated input object (eg, the location of the input object. Angles), it is possible to calculate the three-dimensional position of the input object.

Referring to FIGS. 7A to 7E, for example, in the method of FIG. 14, an eleventh position angle ANG11 is calculated based on the position of a first reference point in the first side image IMAGE1, and the second side image ( In IMAGE2), the 21st position angle ANG21 is calculated based on the position of the second reference point, and the first positional information P_C1 of the first camera device 311, and the second positional information of the second camera device 312 2D position information P_OBJ_XY of the input object OBJ may be generated based on (P_C2), the eleventh position angle ANG11, and the 21st position angle ANG21. In addition, the method of FIG. 14 calculates the twelfth position angle ANG12 based on the position of the first reference point in the first side image IMAGE1, and calculates the position of the second reference point in the second side image IMAGE2. The 22nd position angle ANG22 is calculated, and the first position information P_C1 of the first camera device 311, the second position information P_C2 of the second camera device 312, and the twelfth position angle ANG12 ) And depth information P_OBJ_Z of the input object OBJ based on the 22nd position angle ANG22.

The method of FIG. 14 may determine whether a spatial touch input is generated by the input object based on a three-dimensional position of the input object, and generate a touch event (S1150). For example, it is determined whether the depth (that is, the distance from the front of the display device to the input object) among the three-dimensional positions of the input object is within the reference range, and if it is within the reference range, the display device corresponding to the input object It may be determined that a touch input has occurred at one point (or one area). When a function to be executed (eg, an image to be displayed) exists in response to a touch input, the method of FIG. 14 may generate a touch event for executing the function.

16 is a flowchart illustrating another example of a spatial touch sensing method performed in the display device of FIG. 12.

6, 12, 13, and 16, the method of FIG. 16 may pre-process side images, respectively (S1310). Here, the side images may be obtained from the first to fourth camera devices 311 to 314 described with reference to FIG. 12.

Thereafter, the method of FIG. 16 may detect an input object from each of the pre-processed images (ie, pre-processed side images) (S1120).

The method of FIG. 16 may determine whether the input object includes a plurality of sub-objects (S1330). Here, the sub-objects may be separated or separated from each other, and each may correspond to an image that can be recognized as one input object.

When the input object includes only one sub-object, the method of FIG. 16 determines that only one touch input exists, and, as described with reference to FIG. 14, calculates a three-dimensional position of the input object (S1350). , A touch event may be generated based on the 3D location (S1360).

In contrast, when the input object includes a plurality of sub-objects, the method of FIG. 16 may determine that a multi-touch input exists and determine one sub-object as the input object based on the first condition (S1340). . Here, the first condition may be a characteristic common to grouped camera devices, as described with reference to FIG. 13. For example, as described with reference to FIG. 13, an input object adjacent to a specific side of the display device is For determination, it may be a condition for selecting an input object having the largest or smallest position angle.

Referring to FIG. 13, for example, the method of FIG. 16 selects a first input object having the smallest position angle from first and second side images (or preprocessed first and second side images), and , A second input object having the smallest position angle may be selected from the third and fourth side images (or pre-processed third and fourth side images).

Thereafter, the method of FIG. 16 may calculate a 3D position of the determined (or selected) input object (S1350). For example, the 3D position of the first input object may be calculated based on the position angles of the first input object selected from the first and second side images. Similarly, the 3D position of the second input object may be calculated based on the position angles of the second input object selected from the third and fourth side images.

The method of FIG. 16 may determine whether a spatial touch input is generated by the input object based on a three-dimensional position of the input object, and generate a touch event (S1360). For example, the method of FIG. 16 determines whether at least one of a three-dimensional position of a first input object and a three-dimensional position of a second input object satisfies a touch input condition, and generates a touch event based on whether the condition is satisfied. Can be generated.

The embodiments of the present invention have been described above with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. You can understand. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting.

1: display device
10: spatial touch detection module
100: display
130: housing
131: main body
132: top cover
133: side cover
134: support
300: input detection unit
311, 312, 313, 314: first to fourth camera devices
320: infrared light source
500: control unit
700: sensory provision unit

Claims (9)

Display panel;
A first camera device disposed in front of the display panel and generating a first side image of a sensing space from a display surface of the display panel to a preset distance;
A second camera device disposed in front of the display panel and spaced apart from the first camera device, and generating a second side image of the sensing space; And
And a control unit for recognizing a non-contact type spatial touch input within the sensing space based on the first and second side images,
The control unit,
An extractor configured to detect a first input object located in the sensing space from the first and second side images; And
A calculator configured to calculate 3D location information of the first input object based on first location information in the first side image of the first input object and second location information in the second side image,
Further comprising an infrared light source for forming a depth of the sensing space by projecting infrared light into the sensing space,
The angles of view of the first camera device and the second camera device each have 130 degrees to 150 degrees,
The center of the angle of view of the first camera device faces a first direction,
The center of the angle of view of the second camera device faces a second direction crossing the first direction,
The first direction and the second direction each face an area center of the display surface. The method of claim 1, wherein the extraction unit,
Detecting a first object image corresponding to the first input object from the first side image based on a preset pattern,
A first reference point of the first object image is determined based on a predefined feature point in a preset pattern,
Detecting a second object image corresponding to the first input object from the second side image based on a preset pattern,
A display device that determines a second reference point of the second object image based on a predefined feature point in a preset pattern. The method of claim 2, wherein the calculation unit,
Calculate a first position angle based on the position of the first reference point in the first side image, calculate a second position angle based on the position of the second reference point in the second side image, and the first camera A first location calculator that generates two-dimensional location information of the first input object based on first location information of the device, second location information of the second camera device, the first location angle, and the second location angle ; And
Calculate a third position angle based on the position of the first reference point in the first side image, calculate a fourth position angle based on the position of the second reference point in the second side image, and the first camera A second position calculation that generates depth information of the first input object based on the first position information of the device, the second position information of the second camera device, the third position angle, and the fourth position angle Includes wealth,
The first position angle is an angle between one surface of the sensing space and the first input object based on the first camera device,
The second position angle is an angle between the one surface of the sensing space and the first input object based on the second camera device,
The third position angle is an angle between the display panel and the first input object based on the first camera device,
The third position angle is an angle between the display panel and the first input object based on the first camera device,
The 3D location information of the first input object includes the 2D location information and the depth information. The method of claim 2, wherein the extraction unit,
A tracking unit for tracking a change of the first reference point over time; And
A display device comprising: a correction unit correcting the change of the first reference point using a Kalman filter or a low frequency filter. The method of claim 1, wherein the control unit,
A background removal unit configured to generate a first preprocessed image by removing a background image from the first side image;
A noise removing unit for generating first original images by binarizing the first preprocessed images based on a preset threshold; And
Selecting one of the candidate object images as a shadow image based on at least one of sizes of candidate object images included in the first original image, symmetry between the candidate object images, and placement positions of the candidate object images, Further comprising a shadow removal unit for generating a first shadow removal image by removing the shadow image from the first original image,
The first shadow removal image is provided to the extraction unit as the first side image. The method of claim 1, wherein, when the first input object includes a plurality of sub-objects that are separated from each other, the extraction unit selects a first sub-object adjacent to one side of the display panel among the sub-objects as a first input object. Display device to choose from. The method of claim 6,
A third camera device disposed in front of the display panel and spaced apart from the first and second camera devices, and generating a third side image of the sensing space; And
A fourth camera device disposed in front of the display panel and spaced apart from the first to third camera devices, and further comprising a fourth camera device for generating a fourth side image of the sensing space,
The extraction unit detects a second input object located in the sensing space from the third and fourth side images,
The calculation unit calculates 3D position information of the second input object based on third position information in the third side image of the second input object and fourth position information in the fourth side image. The method of claim 1, wherein the extraction unit,
When at least one of the first side image and the second side image is a partial image near the first input object, a shape analysis unit that performs shape analysis on the first input object, and
When at least one of the first side image and the second side image is an entire distant image of the first input object, the display device further comprises a ratio analysis unit configured to perform ratio analysis on the first input object. In a spatial touch sensing method performed in a display device including first and second camera devices disposed to be spaced apart from each other in front of a display panel,
Generating first and second side images of a sensing space from a display surface of the display panel to a preset distance through the first and second camera devices;
Detecting an input object located in the sensing space from the first and second side images; And
Comprising the step of calculating 3D location information of the input object based on the first location information in the first side image of the input object and the second location information in the second side image,
In the step of generating the first and second side images, an infrared light source projects infrared light onto the sensing space to form a depth of the sensing space,
The angles of view of the first camera device and the second camera device each have 130 degrees to 150 degrees,
The center of the angle of view of the first camera device faces a first direction,
The center of the angle of view of the second camera device faces a second direction crossing the first direction,
The first direction and the second direction are respectively directed toward the center of an area of the display surface.

Images