KR102125756B1 - Appratus and method for intelligent vehicle convenience control - Google Patents

Abstract

The present invention relates to an intelligent vehicle convenience control device and method for actively controlling a convenience device or a safety device in a vehicle and providing a customized service to a passenger. An intelligent vehicle convenience control device according to an embodiment includes one or more images in a vehicle. An image information acquisition unit for acquiring information; An occupant information learning unit that recognizes an occupant in a vehicle from the acquired image information and analyzes the occupant's status information and behavior patterns; And a convenience device controller that extracts environment information in the vehicle and determines whether a service is provided based on the learned occupant information and controls the convenience device.

Description

Intelligent vehicle convenience control device and method{APPRATUS AND METHOD FOR INTELLIGENT VEHICLE CONVENIENCE CONTROL}

The present invention relates to an intelligent vehicle convenience control device and method, and more particularly, to an intelligent vehicle convenience control device and method for actively controlling a convenience device or safety device in a vehicle and providing a customized service to a passenger.

Korean Patent Publication No. 10-2012-0024225 relates to a car seat auto-adjustment system, which acquires the physical condition of the occupant of the vehicle as an image capturing unit and uses this image information to take into account safety and convenience when driving a vehicle. After automatically deriving the optimal position of the vehicle, the vehicle seat is automatically adjusted to the optimal state, so that the vehicle seat auto-adjustment system that adapts to the passenger's body condition using image information that can further improve the safety and convenience of the occupant in the vehicle. It is about.

However, the prior art only discloses the contents of one service for automatic adjustment of the car seat, which is different from the intelligent vehicle convenience control device according to the vehicle environment information to be provided in the present invention.

According to the present invention, an intelligent vehicle convenience control device may be implemented through deep learning by analyzing in-vehicle occupant information and indoor/outdoor environment information and providing a customized convenience service to in-vehicle occupants.

According to one embodiment, the intelligent vehicle convenience control apparatus of the present invention, the image information acquisition unit for acquiring one or more image information in the vehicle, recognizes the occupant in the vehicle from the acquired image information and analyzes the occupant's status information and behavior patterns It may include a passenger information learning unit, and a convenience device control unit for extracting environmental information in the vehicle and determining whether to provide a service based on the learned passenger information and controlling the convenience device.

The image information acquisition unit is disposed in the vicinity of the room mirror in the vehicle to acquire image information of an object in the front and rear seats, and the image information acquisition unit may include one or more of an RGB camera, a 3D depth sensor and a thermal IR sensor. .

The occupant information learning unit may learn at least one of the occupant information including the driver, gender, and body type information from one or more image information as occupant information.

The occupant information learning unit may recognize the face and contour of the occupant in the vehicle from one or more image information and estimate the occupant's state information and behavior patterns through a deep learning model.

The deep learning model may further include calculating the skeleton information of the occupant and determining an input signal of the occupant based on the calculated skeleton information of the occupant.

In addition, the deep learning model extracts the feature information and contour information of the occupant from one or more image information to analyze the occupant's condition information, wherein the occupant's condition information includes clothing information worn by the occupant, body temperature estimation information of the occupant, and the occupant It may include one or more of the driver's driving concentration.

In addition, the deep learning model determines the personalized service of the occupant based on the learned behavior pattern information of the occupant, where the occupant behavior pattern information is information on the content reproduction preference of the occupant, the destination pattern information of the occupant, the occupant's driving habits, the occupant It may include one or more of the service control command.

The convenience device control unit may control one or more occupant customized services among media device control in the vehicle, seat arrangement adjustment, air conditioner operation determination, air conditioning device control, airbag deployment pressure control, driver notification, and vehicle operation control.

The present invention can implement an optimized convenient device control process in a vehicle using relatively low-cost cameras or sensors such as RGB cameras, depth sensors, and thermal IR sensors.

In addition, based on the video information in the vehicle, through the analysis of information about the passengers besides the driver, the accuracy of implementation for the customized service required for the vehicle is increased, and deep learning for each one or more occupants also enables the occupants of vehicles used by multiple members. We can provide customized vehicle convenience services.

Further, according to the present invention, occupant information, status information, and behavior patterns can be systematically upgraded through deep learning learning and analysis, and can be upgraded through software simple processing.

According to the present invention, it is possible to provide a more appropriate in-vehicle convenience service by determining the current state of the occupant and the intention of the occupant, such as the recognition of the occupant and the analysis of the input signal of the occupant according to the skeleton information of the occupant.

According to the present invention, it is possible to appropriately control the indoor environment (air conditioner, air conditioner) in the vehicle through the clothes information and body temperature estimation information of the occupant.

According to the present invention, a deep learning model may be implemented in an external cloud server to reduce computing power problems and cost burden in the vehicle.

1 is a block diagram of an intelligent vehicle convenience control device according to an embodiment of the present invention.
2A to 2C are exemplary embodiments in which the image information acquisition unit of the present invention is disposed in the vicinity of a room mirror in a vehicle.
3 is a conceptual diagram of a seat space in a vehicle recognizable by the image information acquisition unit of the present invention.
4 is an example of one or more image information acquired by the image information acquisition unit of the present invention.
5 and 6 are examples of the deep learning model of the present invention extracting the skeleton information of the occupant and analyzing the behavior pattern and input information of the occupant from the extracted skeleton information coordinates.
7 is an example of a process of training the deep learning model of the present invention.
8 is an example of a conceptual diagram for a detailed configuration of a convenience device in a vehicle.

Since the description of the present invention is merely embodiments for structural or functional description, the scope of the present invention should not be interpreted as being limited by the embodiments described in the text. That is, since the present embodiments can be variously modified and have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing technical ideas.

In addition, the purpose or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such an effect, and the scope of the present invention should not be understood as being limited thereby.

In addition, the accompanying drawings are provided to help understanding of the present invention, and provide embodiments with detailed description. However, the technical features of the present invention are not limited to specific drawings, and the features disclosed in each drawing may be combined with each other to form a new embodiment.

The apparatus for providing passenger comfort disclosed in the following embodiments will be described in more detail with reference to each drawing.

1 is a block diagram of an intelligent vehicle convenience control device 100 according to an embodiment of the present invention. According to an embodiment, the intelligent vehicle convenience device 100 may include an image information acquisition unit 110, a passenger information learning unit 120, a convenience device control unit 140, and a communication unit 130.

As an example, the image information acquisition unit 110 may acquire one or more image information in a vehicle. At this time, the image information acquisition unit 110 may include an RGB camera, a depth sensor and a thermal IR sensor. However, various other types of cameras, for example, a monocular (mono) camera, a binocular (stereo) camera, an infrared camera, one or more cameras of a lidar, and an LRF sensor may be utilized, and one including a plurality of cameras described above It can be mounted in the vehicle as a module.

At this time, the image information acquiring unit 110 may be disposed near the room mirror in the vehicle to acquire image information obtained by photographing objects in the front and rear seats. 2A to 2C are exemplary embodiments in which the image information acquisition unit of the present invention is disposed in the vicinity of a room mirror in a vehicle.

3 is a conceptual diagram of a seat space in a vehicle that can be recognized by the image information acquisition unit 110 of the present invention, and when the image information acquisition unit 110 according to an embodiment is disposed near a room mirror, the center of the vehicle There is an advantage in that you can check whether a person is boarding for each seat from a single image by photographing both the driver's seat, the passenger seat, and the back seat.

Here, FIG. 4 is an example of one or more image information acquired by the image information acquisition unit of the present invention. The above-described RGB camera, depth sensor, and thermal sensor can acquire image information, respectively, and the image information acquisition unit 110 of the present invention preprocesses and filters each image information to color or contour information, depth information, and temperature. Information, etc. can be obtained.

Referring to FIG. 4, the image information acquisition unit 110 corresponds to a driver's seat, a passenger seat, a rear seat (left), and a rear seat (right) from image information taken using a thermal sensor or an infrared camera, or a thermal imaging camera, etc. Temperature information for the area may be obtained, and the occupant information learning unit 120 may determine whether a person in the vehicle is on board or not, from the temperature information. Furthermore, the occupant information learning unit 120 compares the temperature information of the vehicle inside the image, the temperature information of the occupant, and the characteristic information of the occupant through a deep learning model, thereby estimating body temperature, the thickness and type of clothes the occupant wears, and the like. Information can be learned. As another example, the thickness of the garment may be estimated from a color difference of RGB image information or a depth (volume) value of depth image information.

On the other hand, the intelligent vehicle convenience providing device 100 may further include a communication unit 130 for transmitting and receiving learning data set and deep learning results to an external server. At this time, the external server may be a cloud server. The communication unit 130 may be connected to an external cloud server using a wired/wireless communication network, or transmit/receive a learning data set and deep learning results through a Bluetooth connection with a smartphone or the like capable of in-vehicle communication. In this case, the actual analysis and judgment of the occupant may be made by an external deep learning model.

The deep learning model may pre-process one or more image information acquired by the image information acquisition unit 110 to extract basic data necessary for occupant analysis.

According to another embodiment, the occupant information learning unit 120 may recognize the occupant in the vehicle from the acquired image information and analyze the occupant's status information and behavior patterns. For example, the occupant information learning unit 120 may learn one or more of occupant information including a driver, gender, and body type information from one or more image information as occupant information. For example, the occupant information learning unit 120 determines whether the occupant has boarded the driver's seat, the auxiliary seat (passenger seat), the rear seat (left), and the rear seat (right) from the temperature information obtained from the image information acquisition unit 110, etc. Whether or not it is possible to determine whether or not the occupant in the vehicle, the occupant's gender, age group, and the like, from the results of face recognition and skeletal information extraction for the occupant can learn information related to the occupant.

For example, when identifying the occupant based on the face, the occupant's characteristic information may correspond to the contour of the face, eyes, nose, and mouth distance, and when identifying the occupant based on the body type, the occupant's characteristic information is depth Values or volume values or skeletal information may be applicable.

As another example, the intelligent vehicle convenience control device 100 may change the airbag setting to adaptively change the airbag deployment pressure, direction, etc. when an occupant is a child.

5 and 6 are examples of the deep learning model of the present invention extracting the skeleton information of the occupant and analyzing the behavior pattern and input information of the occupant from the extracted skeleton information coordinates.

For example, the occupant information learning unit 120 may recognize the occupant's face and contour from one or more image information and estimate the occupant's state information and behavior patterns through a deep learning model. For example, the deep learning model may calculate the occupant's skeleton information by extracting the occupant's joint position from the image information.

Here, the joint position may refer to a 2D or 3D position coordinate for each type of joint, and the skeleton information may be expressed as a joint position (coordinate) at each end of each bone, so that the joint type and joint position (coordinate) of the joint at both ends of the skeleton are based. The skeletal information can be calculated. However, the skeletal information may be expressed in various ways, such as the length or angle of the skeleton connecting the joints at both ends of the skeleton in addition to the joint position (coordinate).

Referring to FIG. 5, the occupant driving the vehicle while holding the handle A has shoulder joints (J1, J4), elbow joints (J2, J5), knee joints (J8, J11), ankle joints (J9, J12), etc. It can include the joints of the deep learning model, upper arm bones (B1, B2) by connecting the shoulder joints (J1, J4) and elbow joints (J2, J5) at both ends of the upper arm bones (B1, B2). Skeleton (skeleton) information can be calculated from the joint position.

Referring to FIG. 6, the joint position corresponds to three-dimensional position coordinates (X, Y, Z) for each type of joints J1 to J14. Here, X and Y are the coordinate values of the X and Y axes of the pixel where each joint is located in one frame of visible image information, and Z is the pixel where each joint is located in the frame of depth image information corresponding to the same time. It may be a depth value of.

If the type of joints at both ends of each skeleton constituting the occupant's skeleton is known, the joint positions (coordinates) at both ends of each skeleton can be known by using three-dimensional position coordinates for each type of joints J1 to J14. Skeleton information can be calculated through the joint position (coordinate) for each joint type.

On the other hand, in another embodiment, when the image information acquisition unit 100 does not include a depth camera, the deep learning model calculates the skeleton information by calculating the joint position corresponding to the 2D position coordinates (X, Y) from the RGB image information. Can.

As another embodiment, the deep learning model may calculate skeletal information composed of two-dimensional joint positions (coordinates) and then change to a three-dimensional joint position (coordinates) using a second (sub) deep learning model.

When calculating the skeleton information of the occupant, it is possible to determine and learn various information about the occupant from the skeleton information. The deep learning model can learn the behavior pattern of the occupant as to whether the occupant is driving or what action the occupant performs with input information.

For example, the deep learning model compares the behavioral pattern when the occupant is focused on driving, the case where the occupant has a lot of movement, or the occupant's face recognition results in the occupant's eyes closed for a certain period of time, etc. Can be analyzed. If the occupant information learning unit 120 determines that the current occupant's status information is drowsiness, the convenience device control unit 140 may start a notification service or the like to wake the occupant.

In addition, the deep learning model may analyze the state information of the occupant by extracting the feature information and contour information of the occupant from one or more image information, wherein the occupant's state information includes clothing information worn by the occupant and body temperature estimation information of the occupant And it may include one or more of the driver's driving concentration among the occupants.

Another erotic, deep learning model determines the personalized service of the occupant based on the behavior pattern information of the previously learned occupant, and the behavior pattern information of the occupant includes the occupant's content reproduction preference information, the occupant's destination pattern information, the occupant's driving habits, It may include one or more of the passenger's service control command.

)의 RGB 값, 깊이 값 또는 온도 값이 입력되고 출력층에 k개의 뉴런을 구비하여 각 뉴런에는 k개의 분류(

) 중 어떤 분류에 해당하는지를 판단할 수 있는 정보가 출력될 수 있다. (1<=i<=n, 1<=j<=m, 1<=l<=k)7 is an example of a process of training the deep learning model of the present invention. The deep learning model according to an embodiment includes n*m neurons in the input layer, and each neuron includes pixels in an image of n rows and m columns (

), RGB values, depth values, or temperature values are input and k neurons are provided in the output layer, so each neuron has k classifications (

) May be output to determine which classification. (1<=i<=n, 1<=j<=m, 1<=l<=k)

)의 RGB 값, 깊이 값 및 온도 값이 모두 입력될 수 있는데 이 때에는 입력은 3*n*m 개의 뉴런을 구비해야 한다.On the other hand, each pixel of the image of n rows and m columns in the input layer of the deep learning model (

), the RGB value, the depth value, and the temperature value can all be input. In this case, the input must have 3*n*m neurons.

는 영상정보 외에 특징정보 또는 골격정보에 해당할 수 있고

는 탑승자, 조작대상 및 조작방법, 운전부주의 상태인지 여부, 탑승위치, 인원수, 어린이나 어른, 의복 두께, 체온, 탑승자, 입력신호를 k개의 분류로 나눈 것에 해당할 수 있다.However, unlike in Figure 7,

May correspond to feature information or skeleton information in addition to image information

May correspond to the passenger, the operation target and the operation method, whether the driver is inattentive, the boarding position, the number of people, the child or adult, the thickness of the garment, the body temperature, the passenger, and the input signal divided into k categories.

According to an embodiment, the convenience device controller 140 may extract environment information in the vehicle and determine whether to provide a service based on the learned passenger information and control the convenience device.

8 is an example of a conceptual diagram for a detailed configuration of a convenience device in a vehicle. Referring to FIG. 8, the vehicle 200 includes airbags 212 and 214, air conditioners 222, 224, 226 and 228, content providing devices 232 and 234, a driving device 240, and a seat 250 ), a driving caution device (not shown), a photographing unit 300, and a passenger convenience providing control device 400. Here, the content providing devices 232 and 234 may provide content such as audio and video, or provide a navigation function, and serve as a driving caution device (not shown) that gives attention to the driver through sound or video. It may be. That is, the navigation or driving caution device may correspond to a content providing device.

The convenience device control unit 140 may control one or more occupant customized services of media device control in the vehicle, seat arrangement adjustment, air conditioning device control, airbag deployment pressure control, driver notification, and vehicle driving control.

For example, when the occupant information learning unit 120 learns only one occupant in the driver's seat as occupant information, the convenience device control unit 140 may turn off the airbag in the passenger seat or turn off the air conditioning device in the rear seat. have.

In addition, the convenience device control unit 140 provides a recommended destination based on a preset pattern of the occupant in the navigation when the occupant is 1, or a recommended restaurant according to the preference of a nearby recommended restaurant or the recognized occupant in the navigation when the occupant is 2 or more. You can provide your back as a recommended destination

As another example, when the occupant information learning unit 120 learns that the occupant in the passenger seat is a child having a small body shape, the convenience device control unit 140 prevents direct wind from being transmitted toward the child when the air conditioner or heater is operated. You can adjust the air-conditioning or heater delivery direction, or change the air-conditioner deployment settings of the passengers to suit your child.

As another example, when the occupant information learning unit 120 learns the type and thickness of clothes worn by the occupant through deep learning, the convenience device control unit 140 compares the proper temperature of the room with the estimating information of the body temperature of the occupant and uses the air conditioner or heater. It is possible to control the air conditioning system such as turning it on/off.

As another example, when the occupant information learning unit 120 calculates the skeleton information of the occupant and learns the posture or position of the occupant, the convenience device control unit 140 may adjust the seat angle of the occupant's seat. At this time, when the occupant information learning unit 120 analyzes the occupant behavior pattern based on the seat angle adjustment value, etc., during which the occupant has previously adjusted through deep learning, the convenience device control unit 130 may determine the seat based on the occupant behavior pattern. The angle can be adjusted. In addition, at this time, the convenience device controller 130 may perform seat angle adjustment in consideration of the type and thickness of clothes worn by the occupant.

As another example, when the occupant information learning unit 120 learns the state of the occupant in a state of driving inattention, the convenience device control unit 140 may provide a notification that alerts the occupant.

On the other hand, according to another embodiment, the intelligent vehicle convenience providing device 100 may further include an interface 150 to interact with the user (not shown), where the interface 150 is occupant through the image analysis of the occupant Input/output can be performed with an input signal input to control a vehicle.

According to another aspect, an intelligent vehicle convenience providing method may be described as follows using the intelligent vehicle convenience providing apparatus 100.

First, the intelligent vehicle convenience providing method includes obtaining one or more image information in a vehicle, recognizing a passenger in a vehicle from the acquired image information, analyzing a state information and behavior pattern of a passenger, and extracting environmental information in the vehicle And determining whether to provide a service based on the learned occupant information, and controlling a convenient device.

In addition, the intelligent vehicle convenience providing method includes the step of recognizing the face and contour of the occupant in the vehicle from the one or more image information and estimating the state information and the behavior pattern of the occupant through the deep learning model, wherein the deep learning model includes one or more images. The feature information and contour information of the occupant are extracted from the information to analyze the occupant's condition information, and the occupant's condition information includes at least one of clothing information worn by the occupant, body temperature estimation information of the occupant, and driver's driving concentration among the occupants It can be a step.

As described above, although described with reference to preferred embodiments of the present application, those skilled in the art variously modify the present application without departing from the spirit and scope of the present invention described in the following claims. And can be changed.

100: intelligent vehicle convenience control device 110: image acquisition unit 120: passenger information learning unit 130: convenience device control unit
140: communication unit 150: interface

Claims (10)

An image information acquisition unit that acquires one or more image information in a vehicle;
A passenger information learning unit recognizing whether a passenger in a vehicle is boarding from the acquired image information and learning the operation of the passenger through deep learning; And
And a convenience device control unit controlling a convenience device including an air conditioning device, an airbag, a seat angle or a navigation device according to information from the image information acquisition unit and the passenger information learning unit,
The occupant information learning unit recognizes the face and contour of the occupant in the vehicle from the one or more image information, calculates the occupant's skeleton information, and learns the occupant's motion through a deep learning model from the calculated occupant's skeleton information coordinates,
In the deep learning model, RGB values or skeletal information of each pixel of the image information are input to an input layer, and information about a passenger, a boarding position, or an operation of the passenger is classified on the output layer,
The convenience device control unit is an intelligent vehicle convenience control device that controls the convenience device based on image information obtained from the image information acquisition unit and results classified by a deep learning model.
According to claim 1,
The image information acquisition unit is arranged in the vicinity of the room mirror in the vehicle to obtain the image information of the front and rear seats of the object,
The image information acquiring unit includes at least one of an RGB camera, a depth sensor and a thermal IR sensor, an intelligent vehicle convenience control device. According to claim 1,
The occupant information learning unit learns one or more of occupant information including the driver, gender, and body type information from the one or more image information as occupant information, an intelligent vehicle convenience control device. delete delete According to claim 1,
The deep learning model extracts feature information and contour information of the occupant from the one or more image information to analyze the occupant's state information,
The state information of the occupant includes at least one of clothing information worn by the occupant, body temperature estimation information of the occupant, and driver's driving concentration among the occupants. According to claim 1,
The deep learning model determines the personalized service of the occupant based on the behavior pattern information of the previously learned occupant,
The behavior pattern information of the occupant includes at least one of the content reproduction preference information of the occupant, the destination pattern information of the occupant, the driving habits of the occupant, and the service control command of the occupant. According to claim 1,
The convenience device control unit is an intelligent vehicle convenience control device for controlling one or more occupant customized services of the in-vehicle media device control, seat arrangement adjustment, air conditioning device control, airbag deployment setting control, driver notification, and vehicle driving control. delete delete

Images