KR20170037585A - Vehicle Center Fascia Control Method Based On Gesture Recognition By Depth Information And Virtual Touch Sensor - Google Patents

Abstract

The present invention relates to a system to control a center fascia of a vehicle, generating a control graphic interface in a center fascia of a vehicle and using a position change of touch coordinates detected in a virtual touch sensor region, and a control method thereof, and more specifically, to an image processing technology to realize a virtual touch sensor region capable of recognizing multi-touch on a graphic interface for integrally controlling the center fascia of the vehicle, to allow a depth camera to recognize and convert one touch motion or multi-touch motion recognized by touch action of a driver in the virtual touch sensor region into depth information, to display a graphic event for control suitable for the depth information on the center fascia of the vehicle, and at the same time, to control the center fascia of the vehicle. According to the present invention, a system to control a center fascia of a vehicle by using depth information of an image and a virtual touch sensor, and a control method thereof installs the depth camera inside the vehicle, sequentially applies an algorithm of recognizing an object by using a flat structure part of the center fascia of the vehicle as a base structure for integrated control, projecting and generating a graphic interface for the integrated control on the flat structure part of the center fascia of the vehicle, and allowing the depth camera to recognize touch motion for a predetermined position of the driver when the driver touches the predetermined position, so as to transfer touch information to a control console. Moreover, the above individual modules are formed in one system.

Description

Technical Field [0001] The present invention relates to a vehicle center fascia control method using depth information of an image and a virtual touch sensor,

The present invention implements a control system for controlling a vehicle center based on a change in position of touch coordinates detected in a virtual touch sensor area and a control graphic interface in a center fascia, A virtual touch sensor area capable of multi-touch recognition on a graphic interface for an integrated control of an automobile center fascia is implemented, and a single touch operation or multi-touch operation recognized by a driver's touch action on the virtual touch sensor area is called a depth camera To the recognition and depth information, and to display appropriate control graphic events on the automobile center fascia, and to control the automobile center fascia.

The center fascia installed inside the car includes a sound module including a wireless radio and a car audio, which are convenience devices for the driver; A display device module including navigation and digital multimedia broadcasting (DMB); An air conditioner module including a radiator and a radiator, and the like. The center fascia is provided on a dash board, is fixed to a bracket and is positioned at the center of the front of the automobile. The driver turns his / her direction toward the center fascia from time to time during driving and extends his / her hands to the sound device module , A display device module, and an air conditioner module. Generally, the sound module and the display module operate a control knob provided on one side of the center fascia to adjust the power or the sound output. The air conditioner module can also control the temperature, wind speed and direction of the air conditioner and the heater by operating a control knob provided at one side of the center pace. However, in the conventional center fascia type, it is inconvenient to manually control a convenience device such as a sound device module, a display device module, and an air conditioner module. In this manual control method, There is a problem in that the risk of accidents is increased because the frontward observation can not be performed continuously.

Similar prior art to disclose depth information of an image and an automobile center fascia control system using the virtual touch sensor and a control method thereof include KR 10-1199144 (B1) registered in the Korean Intellectual Property Office; KR 10-1219933 (B1); KR 10-1446031 (B1); KR 10-1382287 (B1); . The conventional technology does not provide a technique for controlling the automobile center fascia by simultaneously using the depth information and the virtual touch sensor area.

10-1199144 (B1) 10-1219933 (B1) 10-1446031 (B1) 10-1382287 (B1)

The invention aims at meeting the technical needs required from the background of the above-mentioned invention.

It is an object of the present invention to realize a virtual touch sensor area capable of multi-touch recognition on a graphic interface for integrated control of an automobile center fascia and to perform a single touch operation or multi-touch operation recognized by a driver's touch action on the virtual touch sensor area The present invention provides an image processing technique that enables a touch camera to convert a touch operation to recognition and depth information by a depth camera, and displays a suitable control graphic event on the automobile center fascia while controlling the automobile center fascia.

The technical objects to be achieved by the present invention are not limited to the above-mentioned problems, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description will be.

In order to accomplish one object, an automobile center fascia control system and a control method thereof using depth information and a virtual touch sensor according to the present invention include a depth camera installed inside a car and a flat structure part of an automobile center fascia as an infrastructure for integrated control After the object is recognized and the graphic interface for integrated control is projected and generated on the automobile center fascia flat structure, when the driver touches a specific position, the depth camera recognizes the touch operation for the specific position of the driver, An algorithm for transmitting touch information is sequentially applied and the individual modules are configured as one system.

As described above, according to the present invention, it is possible to create a virtual integrated control graphic interface by using the flat structure part of the automobile center fascia as an integrated control base, and to recognize the touch operation to a specific position of the driver in real time, There is an advantage that intuitive control of the automobile center fascia is possible without the adjustment knob for various convenience devices installed in the center fascia.

It is to be understood that the technical advantages of the present invention are not limited to the technical effects mentioned above and that other technical effects not mentioned can be clearly understood by those skilled in the art from the description of the claims There will be.

1 is a flowchart of a depth information processing analysis for implementing an automobile center fascia control system and control method using depth information of an image and a virtual touch sensor according to an embodiment of the present invention;
FIG. 2 is an exemplary view showing an installation position of a depth image capturing module for a virtual touch sensor in an automobile center fascia control system using a virtual touch sensor and depth information of an image according to an embodiment of the present invention; and FIG.
FIG. 3 illustrates an example of a process of correcting coordinates before conversion by one-dimensional linear transformation in an automobile center fascia control system using a depth information of an image, a virtual touch sensor, and a control method thereof according to an embodiment of the present invention;
FIG. 4 is a view illustrating touch judgment in a touch sensor area in an automobile center fascia control system and control method using depth information of an image and a virtual touch sensor according to an embodiment of the present invention; FIG.
FIG. 5 is a diagram illustrating a flow of a touch speed and a locus of a pointer in an automobile center fascia control system using a virtual touch sensor and image depth information according to an embodiment of the present invention.
6 is an exemplary view of an automobile center fascia to be controlled in the automobile center fascia control system using the depth information of the image and the virtual touch sensor and the control method thereof according to the embodiment of the present invention;
FIG. 7 is a flowchart illustrating an automobile center fascia control system using a virtual touch sensor according to an exemplary embodiment of the present invention, and a control method thereof; FIG.
FIG. 8 is a layout diagram of major modules proposed by the automobile center fascia control system using the virtual depth sensor and the virtual touch sensor according to the embodiment of the present invention and its control method; FIG.
FIG. 9 is a view for explaining motion recognition for implementing the automobile center fascia control system using the depth information of the image, the virtual touch sensor, and the control method according to the embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, It is not. In the following description of the present embodiment, the same components are denoted by the same reference numerals and symbols, and further description thereof will be omitted.

The concept of the depth information processing analysis using the depth camera will be described with reference to FIG. As shown in FIG. 1 (a), the detailed configuration of the virtual touch sensor includes a depth image photographing module 11 for obtaining a depth image of a touch sensor area using a touch sensor area; A spatial coordinate correction unit (12) for correcting spatial coordinate distortion of an image with respect to the touch sensor area taken by the depth image photographing module (11); A depth value calculation unit 13 for calculating a representative depth value of the touch sensor region in the depth image whose spatial coordinate distortion is corrected in the spatial coordinate correction unit 12; An object detector 14 for grouping and labeling pixels of a depth image having a depth value different from a representative depth value of the touch sensor area, storing the sequentially stored size values of the objects, and detecting the object; A pointer extracting unit (15) for extracting a pointer to the detected object; And a touch determination unit 16 for determining whether the touch sensor area is touched based on the depth value of the extracted pointer position.

In addition, the order of correction of the spatial coordinates can be made different as shown in Fig. 1 (b). In the case where the correction of the spatial coordinates is performed before the touch determination, as shown in FIG. 1B, the depth image photographing module 21 photographs the touch sensor area to obtain a depth image for the touch sensor area; A depth value calculator 22 for calculating a representative depth value of the touch sensor area in the depth image; An object detection unit 23 for sequentially grouping and labeling pixels of a depth image having a depth value different from a representative depth value of the touch sensor area and storing the depth information as a size value corresponding to the number of objects in order; A pointer extracting unit (24) for extracting a pointer from the detected object; A spatial coordinate correcting unit 25 for correcting spatial coordinate distortion of an image with respect to the touch sensor area; And the spatial coordinate correction order can be changed by the touch determination unit 26 that determines whether or not the touch sensor area is touched based on the depth value of the extracted pointer position.

Here, the spatial coordinate correction is used when the absolute spatial coordinate is applied, and not when the relative spatial coordinate is applied. One image shooting module is installed in any one of the top, bottom, left, right, upper left, lower left, right upper, and lower right areas in a display monitor, screen, flat or curved object used as a virtual touch sensor Or may be installed in more than one place (FIG. 2).

The depth image photographing module 11 can be attached to an external mount such as a hanger or a pedestal, a fixed type inside a wall or a floor, a removable type which is freely attachable and detachable, and a socket type attached to a terminal device. The depth value calculation unit 13 of the touch sensor area calculates a depth value of a specific point by receiving the depth image of the touch sensor area obtained by capturing the touch sensor area by the depth image capturing module 11. The depth value calculation unit 13 is a part that operates only at the beginning of execution and can select the first screen of the depth image. In addition, a screen corresponding to a specific number can be accumulated to obtain a stable image. When the screen is accumulated, the average value is calculated and stored as the representative depth value at each position of the touch sensor. In this case, in addition to the average value, an intermediate value, a minimum value, or the like may be used. When the position or the distance of the photographing module and the touch sensor area is changed, the depth value calculation unit 13 is executed again. The depth value of the touch sensor area is obtained for each screen, and the obtained depth value is compared with the previously stored representative depth value The depth value calculation unit 13 of the touch sensor area is executed again when the absolute difference is larger than the predetermined value.

FIG. 3 is a diagram illustrating that the coordinates before transformation are corrected by one-dimensional linear transformation based on four spatial coordinates that are mapped under the environment in which the depth image capturing module is installed on the left side of the virtual touch sensor area. Is defined.

...수식(1)

... (1)

...수식(2)

... (2)

The corrected post-transformation absolute coordinates can be obtained by using the above equations (1) and (2). Here, x (horizontal) and y (vertical) are coordinates before conversion, and X (horizontal) and Y (vertical) are coordinates after conversion. H _x (width), V _yo , V _y1 (height) correspond to the touch sensor area before conversion, H _X (width) and V _Y (height) correspond to the touch sensor area after conversion. x ₀ (horizontal) and y ₀ (vertical) are the correction offsets.

Each time a new image of the depth image is input, the object detecting unit 14 compares the representative depth value of the touch sensor area stored in the previous step and determines that the object is detected if the two values are out of a predetermined range. That is, if the following equation (3) is satisfied, it is determined that the object is detected in the horizontal direction x and the vertical direction y position, and a new binary image is generated and the pixel value of the corresponding position is stored as '255'.

...수식(3)

... (3)

Here, d is the depth value of the currently input depth image, d _s is the representative depth value of the touch sensor area, and T _i is a specific value, which is set by the user, and is preferably 5 in the millimeter standard.

When object detection is performed for one screen, a binary image having a pixel value of '255' is generated in the object part and '0' is generated in the non-object part. A process of sequentially grouping and labeling pixels having a pixel value of '255' in a binary image is performed. In the labeled order, objects are detected by storing them as '1' for the first object, '2' for the second object, and so on. In addition, noise can be removed through various morphology operations, erosion, expansion, removal, and filling. Since noise may occur in the labeling of the detected object, if the number of pixels of the labeled object is within a certain number, it may be excluded from the labeling object to exclude noise. The pointer extracting unit 15 extracts a pointer from the detected object, and the pointer obtains a position closest to the touch sensor area and uses the pointer as a pointer. The extracted pointer may be a human finger or an indicator. Finding a position close to the touch sensor area depends on the installation direction of the image sensing module. When the depth imaging module is installed on the left side of the touch sensor area, the position of the leftmost pixel in the object is used as a pointer based on the direction of the depth sensor image of the touch sensor area, and when the imaging module is on the right side, A pixel located on the upper side when the image is on the upper side and a pixel located on the lower side when the lower side is used as a pointer.

FIG. 4 is a block diagram showing touch judgment in a touch sensor area, and FIG. 5 is a block diagram showing a flow of a touch speed and a locus of a pointer. The touch determination unit 16 determines whether or not the pointer approaches or touches the touch sensor. Whether or not the touch sensor area is judged to be touched even when the hand or the pointer touches the touch sensor area is determined to be touched or when the touch point is within a certain distance, it can be determined according to the convenience of the user. The touch determiner 16 in the, given that touch the case of contact value of the depth of the pointer d _p (x, y) and the value of the depth at which the position in the depth image of the touch sensor area d _s (x, y ), And determines that the touch is made when the absolute difference between the two values is less than the specific value. That is, when the following equation (4) is satisfied, it is determined that the pointer touches the touch sensor area in the horizontal direction x and the vertical direction y position.

...수식(4)

... (4)

Here, T _D is set by the user as a specific value, and in the case of the millimeter standard, generally 5 is suitable. In this case, in order to prevent malfunctioning of the judgment, the average value of the x-1, y, y + 1 three-dimensional depth values in several neighboring horizontal directions rather than one position in the horizontal direction x and the vertical direction y may be used. Alternatively, neighboring pixel values on the left diagonal, right diagonal, etc. may be used. When it is judged that the approach is touched, the position at which the touched position is compared with the position at which the depth value is compared is different. That is, when the image pickup module is installed on the left side of the touch sensor area, d _p (x, y) which is the depth value of the pointer position and d _s (xT, y), and judges that the touch is made when the difference between the two values is less than the specified value. At this time, the touched position is the position of the horizontal direction xT and the vertical direction y, and if the following equation (5) is satisfied, it is determined that the pointer touches the touch sensor area at the corresponding position.

...수식(5)

(5)

Here, T is set by a user as a specific value, and when an approach distance between the pointer and the touch sensor area is 1 cm, five pixels are suitable. The average value of several neighboring depth values may be used rather than one position of the pointer. When the image capturing module is installed on the right side of the touch sensor area, d _p (x, y) is compared with d _s (x + T, y) it indicates, d _p (x, y) and d _s Compare (x, yT) (x, yT) , and determines the touch whether or not the position, when the bottom of d _p (x, y) and d _s (x, y + T) to judge whether or not the position of (x, y + T) is touched. In addition, the horizontal / vertical depth value of the pointer can be obtained for each screen as shown in FIG. That is, the velocity and direction of the pointer in the horizontal direction, the velocity and direction in the vertical direction, and the distance from the photographing module or the touch sensor area can be known based on the frame rate of the screen. One of the methods for detecting the moving direction and speed of the pointer is to continuously compare the previous frame and the current frame of the depth image acquired from the depth image capturing module to detect the moving direction, the moving speed, and the moving distance of the pointer Can be used.

The depth information of the image according to the present invention and the automobile center fascia control system using the virtual touch sensor and the control method thereof are obtained by installing the depth camera in the automobile and connecting the flat structure part 1 of the center fascia as shown in FIG. The depth camera recognizes the touch operation to a specific position of the driver, recognizes the object by the base, and projects and generates a graphic interface area for virtual integrated control with a video transmission module such as a beam projector, To which the touch information is transmitted. Referring to FIG. 7,

(S100) of selecting the flat structure part of the chain center pacea based on the graphic interface for integrated control to perform the depth camera photographing and the depth image data acquisition and to start the depth image photographing;

A step (S200) of projecting and generating a graphic interface image for integrated control on the flat structure part of the center pace in the step S100;

(S210) extracting touch area background data from the integrated control graphic interface image projected and generated on the flat structure part of the center fascia upon completion of step S200;

An object recognizing step (S300) of recognizing a driver's hand with a depth camera to acquire depth information on the displacement between the graphic interface image for integrated control and the driver's hand projected and generated in step S200;

In operation S400, a touch position is detected by a depth camera to determine which position of the graphic interface image for integrated control has been touched by the driver's hand whose object recognition is completed.

Converting the touch position detected in step S400 into depth information (S500);

A control data synchronization step (S510) of confirming and matching the touch position converted into the depth information in step S500 and the control information embedded on the control console;

(S600) transmitting a code in which the depth information on the touch position and the control data are synchronized to the control console in step S510;

(S610) selecting and extracting the code transmitted in the step S600 as the corresponding control operation information embedded in the control console;

(S700) of performing feedback to transmit the control job information extracted in the step S610 back to the graphic interface for integrated control and reflecting the control result.

8 is a schematic diagram of an overall system 100 for implementing an automobile center fascia control system and a control method based on motion recognition using depth information and a virtual touch sensor according to the present invention.

A depth camera module (110) for acquiring depth information;

A beam projector image sending module 130 for projecting the graphic interface for integrated control to the flat structure 120 of the center pacey;

A control console module 140 in which graphic interface data for integrated control for touch location recognition, control data, and control announcement voice data are embedded;

And a speaker module 150 for outputting a control announcement voice.

The short-range wireless communication module 160 may be a low-power-based short-range wireless communication technology standard (hereinafter referred to as " short-range wireless communication technology "), It is preferable to use a bluetooth communication method.

FIG. 9 is a view for explaining motion recognition for implementing the automobile center fascia control system using the image depth information, the virtual touch sensor, and the control method according to the embodiment of the present invention. When a single touch operation or a multi-touch operation is performed on the virtual touch sensor area 12r implemented on the flat structure part 120 of the center pacea, the depth camera recognizes the coordinates information 10t and 11t of the touch, And transmits the generated control graphic event to the control console module 140 so that the control graphic event suitable for the control graphic event is displayed on the flat structure 120 of the center pascia, thereby realizing a motion recognition based automobile center fascia control system and control method.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, will be. Accordingly, the true scope of the present invention should be determined only by the appended claims.

10t, 11t: coordinate information of the corresponding touch
12r: Virtual touch sensor area
100: Whole system
120: Flat structure of center fascia
140: Control console module

Claims (1)

A depth image photographing module 11; A spatial coordinate correcting unit 12, an image depth value calculating unit 13; An object detecting unit 14; A pointer extracting unit (15) for extracting a pointer to the detected object; A touch determination unit (16) for determining whether or not the touch sensor area is touched; A beam projector image sending module 130 for projecting the graphic interface for integrated control to the flat structure 120 of the center pacey; A control console module 140; A speaker module 150; A method of controlling an automotive center fascia using depth information of an image composed of a short range wireless communication module (160) and a virtual touch sensor,
A step of selecting a flat structure part of a chain center fascia based on a graphic interface for integrated control and starting a depth image photographing (S100);
A step S200 of projecting and generating an integrated control graphic interface image on the flat structure part of the center pacecia;
Extracting touch area background data from the integrated control graphic interface image (S210);
An object recognition step (S300) of recognizing a driver's hand with a depth camera in order to acquire depth information on the displacement between the graphic interface image for integrated control and the driver's hand;
A step (S400) of detecting a position touched by the driver's hand on the graphic interface image for integrated control;
Converting the detected touch position into depth information (S500);
A data synchronization step (S510) of matching the converted depth information with the control information embedded in the control console;
When the control data is not synchronized, returning to the step of detecting the touch position again (S400);
If the control data is synchronized, transmitting the synchronized data code to the control console (S600);
Extracting the transmitted code as the corresponding control operation information embedded in the control console (S610);
Returning to the touch position detection step (S400) if the control operation information in the control console is not extracted (S610);
(S700), when the control task information in the control console is extracted (S610), the feedback for retransmitting the extracted control task information to the graphic interface for integrated control is performed and the control result is reflected Depth information of video and automobile center fascia control method using virtual touch sensor.

Images