CN111860312A - Driving environment adjusting method and device - Google Patents
Driving environment adjusting method and device Download PDFInfo
- Publication number
- CN111860312A CN111860312A CN202010697463.3A CN202010697463A CN111860312A CN 111860312 A CN111860312 A CN 111860312A CN 202010697463 A CN202010697463 A CN 202010697463A CN 111860312 A CN111860312 A CN 111860312A
- Authority
- CN
- China
- Prior art keywords
- processing
- image
- adjusting
- automobile
- passenger
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000006399 behavior Effects 0.000 claims abstract description 80
- 238000012545 processing Methods 0.000 claims abstract description 64
- 230000007613 environmental effect Effects 0.000 claims abstract description 24
- 210000000988 bone and bone Anatomy 0.000 claims abstract description 21
- 230000001105 regulatory effect Effects 0.000 claims abstract 2
- 238000001514 detection method Methods 0.000 claims description 32
- 238000004422 calculation algorithm Methods 0.000 claims description 20
- 238000012549 training Methods 0.000 claims description 16
- 238000003066 decision tree Methods 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 26
- 238000005070 sampling Methods 0.000 description 18
- 230000009471 action Effects 0.000 description 17
- 238000004590 computer program Methods 0.000 description 9
- 238000009826 distribution Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000007704 transition Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 239000000126 substance Substances 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000001816 cooling Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000010438 heat treatment Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000004243 sweat Anatomy 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60H—ARRANGEMENTS OF HEATING, COOLING, VENTILATING OR OTHER AIR-TREATING DEVICES SPECIALLY ADAPTED FOR PASSENGER OR GOODS SPACES OF VEHICLES
- B60H1/00—Heating, cooling or ventilating [HVAC] devices
- B60H1/00642—Control systems or circuits; Control members or indication devices for heating, cooling or ventilating devices
- B60H1/00735—Control systems or circuits characterised by their input, i.e. by the detection, measurement or calculation of particular conditions, e.g. signal treatment, dynamic models
- B60H1/00742—Control systems or circuits characterised by their input, i.e. by the detection, measurement or calculation of particular conditions, e.g. signal treatment, dynamic models by detection of the vehicle occupants' presence; by detection of conditions relating to the body of occupants, e.g. using radiant heat detectors
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60H—ARRANGEMENTS OF HEATING, COOLING, VENTILATING OR OTHER AIR-TREATING DEVICES SPECIALLY ADAPTED FOR PASSENGER OR GOODS SPACES OF VEHICLES
- B60H1/00—Heating, cooling or ventilating [HVAC] devices
- B60H1/00642—Control systems or circuits; Control members or indication devices for heating, cooling or ventilating devices
- B60H1/00814—Control systems or circuits characterised by their output, for controlling particular components of the heating, cooling or ventilating installation
-
- E—FIXED CONSTRUCTIONS
- E05—LOCKS; KEYS; WINDOW OR DOOR FITTINGS; SAFES
- E05F—DEVICES FOR MOVING WINGS INTO OPEN OR CLOSED POSITION; CHECKS FOR WINGS; WING FITTINGS NOT OTHERWISE PROVIDED FOR, CONCERNED WITH THE FUNCTIONING OF THE WING
- E05F15/00—Power-operated mechanisms for wings
- E05F15/70—Power-operated mechanisms for wings with automatic actuation
- E05F15/71—Power-operated mechanisms for wings with automatic actuation responsive to temperature changes, rain, wind or noise
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Thermal Sciences (AREA)
- Mechanical Engineering (AREA)
- Traffic Control Systems (AREA)
Abstract
The application discloses a method and a device for regulating a driving environment, wherein the method and the device are used for obtaining a plurality of bone key points of a driver and passengers by processing images of the driver and passengers in a vehicle; obtaining passenger behaviors of the driver and the passengers according to a plurality of skeleton key points; on the basis, the driving environment of the automobile is adjusted according to the passenger behaviors and the environmental parameters, so that the driver can obtain a satisfactory environment in the automobile without any operation. Compared with the situation that only the environment can be adjusted based on the request of the user in the current intelligent cabin, the environment can be adjusted based on the behavior of the passenger, so that good care can be automatically provided for drivers and passengers, and the driving experience of the passenger is improved.
Description
Technical Field
The application relates to the technical field of vehicles, in particular to a driving environment adjusting method and device.
Background
In the trend of automobile product intellectualization, intelligent cabin design is the key point of competition of automobile manufacturers at present, and the automobile sales volume is improved by means of more intelligent cabin design and better driving experience provided for passengers, so that better economic benefit is obtained.
The inventor of the application finds in practice that in the design of the intelligent cockpit of each current large automobile manufacturer, only the environment or feedback matched with the request can be provided for the passenger based on the request of the user, but the requirement of the passenger cannot be automatically met, so that the driving experience of the passenger is poor.
Disclosure of Invention
In view of this, the present application provides a method and an apparatus for adjusting a driving environment, which are used to automatically provide a good care for a driver, so as to improve the driving experience of a passenger.
In order to achieve the above object, the following solutions are proposed:
a driving environment adjusting method is applied to an automobile, and comprises the following steps:
acquiring images of drivers and passengers in the automobile by utilizing the cameras arranged in the automobile at multiple angles;
processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers;
processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of the driver and passengers;
and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters.
Optionally, the image includes a visible light image and/or an infrared light image.
Optionally, the processing the image by using the bone detection model includes:
processing the image by utilizing a skeleton detection model based on an HRnet network;
or processing the image by utilizing a skeleton detection model based on an OpenPose network.
Optionally, the adjusting the driving environment of the automobile according to the riding behavior and the environmental parameters includes:
processing the riding behaviors and the environmental parameters based on a care service decision tree to obtain adjustment parameters;
and adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
Optionally, the method further comprises the steps of:
and carrying out model training based on a Baum _ Welch algorithm to obtain the passenger behavior recognition model.
A riding environment adjusting device applied to an automobile, the riding environment adjusting device comprising:
the image acquisition module is used for receiving images of drivers and passengers in the automobile acquired by the cameras arranged in the automobile at multiple angles;
the first processing module is used for processing the image by utilizing a skeleton detection model and extracting a plurality of skeleton key points of the driver and passengers;
The second processing module is used for processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of the driver;
and the adjusting execution module is used for adjusting the driving environment of the automobile according to the passenger behavior and the environmental parameters.
Optionally, the image includes a visible light image and/or an infrared light image.
Optionally, the first processing module includes:
the first processing unit is used for processing the image by utilizing a skeleton detection model based on an HRnet network;
or the second processing unit is used for processing the image by utilizing a skeleton detection model based on an OpenPose network.
Optionally, the adjusting execution module includes:
the environment calculation unit is used for processing the riding behaviors and the environment parameters based on a care service decision tree to obtain adjustment parameters;
and the parameter adjusting unit is used for adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
Optionally, the method further includes:
and the model training module is used for carrying out model training based on a Baum _ Welch algorithm to obtain the passenger behavior recognition model.
A storage medium having stored thereon program code that, when executed, implements the steps of the ride environment adjustment method described above.
According to the technical scheme, the application discloses a driving environment adjusting method and device, and particularly relates to a method and device for acquiring images of drivers and passengers in a vehicle by utilizing a camera arranged in the vehicle at multiple angles; processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers; processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of a driver; and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters. Compared with the situation that environment adjustment can be carried out only based on the request of the user in the current intelligent cabin, the passenger behavior monitoring system can automatically monitor the passenger behavior of the driver and the passenger and adjust the environment, so that good care is automatically provided for the driver and the passenger, and the driving experience of the passenger is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for adjusting a driving environment according to an embodiment of the present application;
fig. 2 is a schematic diagram of a network structure of HRNet;
FIG. 3 is a schematic illustration of a plurality of skeletal keypoints according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a skeleton key point detection principle based on an OpenPose network; (ii) a
FIG. 5 is a schematic diagram of a hidden Markov algorithm based passenger behavior recognition model;
FIG. 6 is a schematic diagram of a care decision tree according to an embodiment of the present application;
FIG. 7 is a block diagram of a riding environment adjusting apparatus according to an embodiment of the present application;
fig. 8 is a block diagram of a controller of an automobile according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Example one
Fig. 1 is a flowchart of a driving environment adjusting method according to an embodiment of the present application.
As shown in fig. 1, the driving environment adjusting method provided in this embodiment is applied to an automobile for adjusting a driving environment in the automobile, and the specific adjusting method includes the following steps:
And S1, collecting multi-angle images of drivers and passengers in the automobile.
The automobile comprises a passenger cabin, wherein a plurality of cameras with different angles are arranged in the passenger cabin in the automobile, and image data output by each camera is acquired through a data interface, so that multi-angle images of drivers and passengers are obtained.
The camera here may be a visible light camera for visible light based or an infrared camera based on infrared light. This embodiment can also gather the image in order can be when the light is darker in the car, has installed visible light camera and infrared camera simultaneously to obtain visible light image and infrared image, the preferred near infrared camera of infrared camera in this application.
And S2, extracting a plurality of skeletal key points of the driver and the passengers.
Specifically, the visible light image and/or the infrared image are processed by utilizing a skeleton detection model, so that a plurality of skeleton key points of corresponding drivers and passengers are obtained.
In order to remove the interference of unimportant image information such as ornaments, hair style, etc., the present embodiment uses a skeleton detection model based on HRNet to process the image, and extracts a plurality of skeleton key points of the driver and passengers in the image, wherein the network structure of HRNet is shown in fig. 2. In addition, a plurality of bone key points can be obtained by utilizing a bone detection model based on an openpos network, as shown in fig. 3.
The blocks in fig. 2 represent feature maps, the horizontal arrows represent convolution operations, the upward arrows represent up-sampling operations, and the downward arrows represent down-sampling operations. The network divides the input image into three groups of characteristic extraction branches, extracts image characteristics from different resolution levels, and performs information exchange and fusion among the branches in the characteristic extraction process, namely, the output characteristic layers of different branches are combined together through convolution operation to perform subsequent convolution operation.
For example, inputting a picture of a person, firstly carrying out size conversion operation, then carrying out convolution mathematical operation, then carrying out direct copying and down sampling in parallel, and carrying out convolution on two groups of results respectively, then carrying out up-sampling and down-sampling on the two groups of results, then carrying out copying and down-sampling, then carrying out convolution, up-sampling and down-sampling, next carrying out convolution kernel up-sampling, and finally carrying out convolution to obtain the positions of visible key points in the image, and finally obtaining the coordinate information of 17 skeletal key points of each passenger:
S＝H(I)
wherein, S represents the output skeleton key point coordinate distribution graph, H represents HRNet or OpenPose network, and I represents the original image.
OpenPose is a bottom-up skeletal key point detection depth neural network, namely, all skeletal key points in an image are extracted first, and then people to which the key points belong are identified. Inputting a picture, extracting features through a convolution network to obtain a group of feature maps, then dividing the feature maps into two sub-networks, and respectively extracting a part confidence map and a part association map by using a CNN (convolutional neural network). On the basis, the final part relationship is obtained by using an even matching method, and key points of the same person are connected as shown in fig. 4. Wherein S is a skeleton key point distribution thermodynamic diagram, L represents PAF (partial affinity field diagram), and is a matrix with the same size as the key point distribution thermodynamic diagram, and the data at each point represents the association vector at the point and represents the association degree between the points.
And the even matching is to match the points detected in the key point distribution thermodynamic diagram according to the values in the partial affinity field diagram, and judge which key points belong to the same person. Two by two key points dj1And dj2The degree of matching between is
Wherein the content of the first and second substances,
is the affinity vector at point P on limb c, which can be obtained from the partial affinity field map.
And S3, identifying passenger behavior of the driver based on the plurality of skeletal key points.
And after a plurality of skeleton key points are obtained, inputting the key points into a passenger behavior recognition model based on hidden Markov to be processed to obtain passenger behaviors of corresponding drivers, namely recognizing at least four behaviors of the passengers such as body curling, coat wearing, coat taking off, sweat wiping and the like. Hidden markov modeling of passenger action sequences:
wherein, x is the action sequence of taking a bus that the camera was observed, y is the action sequence that this scheme was waited to discern, and P is discernment probability. Fig. 5 shows a hidden markov model based passenger behavior recognition module, arrows indicate state transitions in the video information, boxes indicate action states to be recognized, and circles indicate observed action states in the video.
And S4, adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters.
After the passenger behavior of the driver is obtained, the driving environment of the vehicle is adjusted based on the corresponding driver (driver or passenger) and the current environmental parameters. The environmental parameters are parameters such as the temperature inside the vehicle, the temperature outside the vehicle, the weather condition of the parking space and the like. The regulation of the driving environment is to control the automobile to perform actions such as heating, cooling, windowing, window closing and the like, and further comprises the setting of the temperature of the automobile if the automobile is heated or cooled and the setting of the opening degree of the window if the automobile is windowed.
Specifically, the operations herein include two parts:
firstly, processing riding behaviors and environmental parameters based on a care service decision tree to obtain adjustment parameters; and then, adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
The care service decision tree is specifically shown in fig. 6, which is generated by the following strategy.
The first step, collecting samples and marking training sets:
D＝{(g1,k1),…,(gm,km)}
wherein, g1,…,gmIs a collected scene sample value, k1,…,kmIs a label for caring service decision, and the attribute set is as follows: l ═ L1,l2,l3And the three state attributes of passenger action, temperature inside the vehicle and temperature outside the vehicle are represented.
And secondly, generating a root node by using all the training set data.
Thirdly, selecting the optimal division attribute L from L*For each value of the attribute, a child node is generated for the node, containing l*And (4) all samples with the attributes of corresponding values, if the sample set is an empty set, the node is a leaf node, otherwise, the decision tree with the node as a following node is taken as a child node, and the child node repeats the third step until the leaf node.
The attribute optimal division index adopted in the third step adopts an information gain rate:
Wherein the content of the first and second substances,
v is the number of values of l. Information gain
Information entropy of set D
Wherein, ratiowIs the proportion of w samples.
According to the technical scheme, the embodiment provides the driving environment adjusting method, and particularly relates to a method for acquiring images of drivers and passengers in a vehicle by utilizing a camera which is arranged in the vehicle at multiple angles; processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers; processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of a driver; and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters. Compared with the situation that environment adjustment can be carried out only based on the request of the user in the current intelligent cabin, the passenger behavior monitoring system can automatically monitor the passenger behavior of the driver and the passenger and adjust the environment, so that good care is automatically provided for the driver and the passenger, and the driving experience of the passenger is improved.
In addition, the method also comprises the step of carrying out model training by adopting a Baum-Welch algorithm so as to obtain the passenger behavior recognition model. The passenger behavior recognition model is obtained by training labeled data based on a hidden Markov algorithm, so that a specific passenger behavior recognition model is obtained, and the specific training process is as follows:
In the first step, a hidden markov model λ ═ a, B, pi ] parameter of the scene is constructed randomly.
And secondly, obtaining expected values of the parameters through forward and backward calculation, wherein forward variables:
αt(i)＝P(x1,…,xn,qt＝yi|λ)
for a given model, the detected passenger bone sequence is x1,…,xnAnd the hidden state at time t is yiJoint probability of (d), backward variable:
βt(i)＝P(xt+1,…,xn|qt＝yi,λ)
for a given model, the hidden state at the time t is yiAnd the subsequently detected bone sequence is xt+1,xt+2….
Taken together, these two variables represent the probability that, given a detected bone sequence, the hidden state is an action class i at time t:
given a bone sequence, the probability of time t going from hidden state i to j:
ξt(i,j)＝P(qt＝yi,qt＝yy|x,λ)
the expectation that the hidden state is i is then:
the expectation that the hidden state transitions from i to j is
Obtaining estimated initial state probabilities in hidden Markov models of a scene
π*＝λ1(i)
Estimating transition probabilities
Estimating output observation probability
And thirdly, updating system parameters according to the system estimation value calculated in the second step.
And fourthly, repeating the second step and the third step until the system parameters are stable.
Example two
Fig. 7 is a block diagram of a driving environment adjustment apparatus according to an embodiment of the present application.
As shown in fig. 7, the driving environment adjusting device provided in this embodiment is applied to an automobile for adjusting a driving environment in the automobile, and the adjusting device specifically includes an image capturing module 10, a first processing module 20, a second processing module 30, and an adjusting execution module 40.
The image acquisition module is used for acquiring multi-angle images of drivers and passengers in the automobile.
The automobile comprises a passenger cabin, wherein a plurality of cameras with different angles are arranged in the passenger cabin in the automobile, and image data output by each camera is acquired through a data interface, so that multi-angle images of drivers and passengers are obtained.
The camera here may be a visible light camera for visible light based or an infrared camera based on infrared light. This embodiment can also gather the image in order can be when the light is darker in the car, has installed visible light camera and infrared camera simultaneously to obtain visible light image and infrared image, the preferred near infrared camera of infrared camera in this application.
The first processing module is used for extracting a plurality of skeletal key points of the driver and passengers.
Specifically, the visible light image and/or the infrared image are processed by utilizing a skeleton detection model, so that a plurality of skeleton key points of corresponding drivers and passengers are obtained. The module includes a first processing unit or a second processing unit.
The first processing unit is used for processing the image by adopting a skeleton detection model based on HRNet to extract a plurality of skeleton key points of the driver and the passenger in the image in order to remove the interference of unimportant image information such as ornaments, hair styles and the like, wherein the network structure of HRNet is shown in fig. 2. The second processing unit is configured to obtain a plurality of bone key points by using the openpos network-based bone detection model, as shown in fig. 3.
The blocks in fig. 2 represent feature maps, the horizontal arrows represent convolution operations, the upward arrows represent up-sampling operations, and the downward arrows represent down-sampling operations. The network divides the input image into three groups of characteristic extraction branches, extracts image characteristics from different resolution levels, and performs information exchange and fusion among the branches in the characteristic extraction process, namely, the output characteristic layers of different branches are combined together through convolution operation to perform subsequent convolution operation.
For example, inputting a picture of a person, firstly carrying out size conversion operation, then carrying out convolution mathematical operation, then carrying out direct copying and down sampling in parallel, and carrying out convolution on two groups of results respectively, then carrying out up-sampling and down-sampling on the two groups of results, then carrying out copying and down-sampling, then carrying out convolution, up-sampling and down-sampling, next carrying out convolution kernel up-sampling, and finally carrying out convolution to obtain the positions of visible key points in the image, and finally obtaining the coordinate information of 17 skeletal key points of each passenger:
S＝H(I)
wherein, S represents the output skeleton key point coordinate distribution graph, H represents HRNet or OpenPose network, and I represents the original image.
OpenPose is a bottom-up skeletal key point detection depth neural network, namely, all skeletal key points in an image are extracted first, and then people to which the key points belong are identified. Inputting a picture, extracting features through a convolution network to obtain a group of feature maps, then dividing the feature maps into two sub-networks, and respectively extracting a part confidence map and a part association map by using a CNN (convolutional neural network). On the basis, the final part relationship is obtained by using an even matching method, and key points of the same person are connected as shown in fig. 4. Wherein S is a skeleton key point distribution thermodynamic diagram, L represents PAF (partial affinity field diagram), and is a matrix with the same size as the key point distribution thermodynamic diagram, and the data at each point represents the association vector at the point and represents the association degree between the points.
And the even matching is to match the points detected in the key point distribution thermodynamic diagram according to the values in the partial affinity field diagram, and judge which key points belong to the same person. Two by two key points dj1And dj2The degree of matching between is
Wherein the content of the first and second substances,
is the affinity vector at point P on limb c, which can be obtained from the partial affinity field map.
The second processing module is for identifying passenger behavior of the occupant based on the plurality of skeletal keypoints.
And after a plurality of skeleton key points are obtained, inputting the key points into a passenger behavior recognition model based on hidden Markov to be processed to obtain passenger behaviors of corresponding drivers, namely recognizing at least four behaviors of the passengers such as body curling, coat wearing, coat taking off, sweat wiping and the like. Hidden markov modeling of passenger action sequences:
wherein, x is the action sequence of taking a bus that the camera was observed, y is the action sequence that this scheme was waited to discern, and P is discernment probability. Fig. 5 shows a hidden markov model based passenger behavior recognition module, arrows indicate state transitions in the video information, boxes indicate action states to be recognized, and circles indicate observed action states in the video.
And the adjusting execution module is used for adjusting the driving environment of the automobile according to the passenger behavior and the environmental parameters.
After the passenger behavior of the driver is obtained, the driving environment of the vehicle is adjusted based on the corresponding driver (driver or passenger) and the current environmental parameters. The environmental parameters are parameters such as the temperature inside the vehicle, the temperature outside the vehicle, the weather condition of the parking space and the like. The regulation of the driving environment is to control the automobile to perform actions such as heating, cooling, windowing, window closing and the like, and further comprises the setting of the temperature of the automobile if the automobile is heated or cooled and the setting of the opening degree of the window if the automobile is windowed.
In particular, the module comprises in particular an environment calculation unit and a parameter adjustment unit.
The environment calculation unit is used for processing the riding behaviors and the environment parameters based on the care service decision tree to obtain adjustment parameters; and the parameter adjusting unit is used for adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
According to the technical scheme, the driving environment adjusting device is provided, and particularly, the images of drivers and passengers in a vehicle are collected by the aid of the cameras arranged in the vehicle at multiple angles; processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers; processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of a driver; and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters. Compared with the situation that environment adjustment can be carried out only based on the request of the user in the current intelligent cabin, the passenger behavior monitoring system can automatically monitor the passenger behavior of the driver and the passenger and adjust the environment, so that good care is automatically provided for the driver and the passenger, and the driving experience of the passenger is improved.
In addition, the passenger behavior recognition model further comprises a magic training module, and the magic training module adopts a Baum-Welch algorithm to perform model training, so that the passenger behavior recognition model is obtained. The passenger behavior recognition model is obtained by training labeled data based on a hidden Markov algorithm, so that a specific passenger behavior recognition model is obtained.
EXAMPLE III
The present embodiment provides an automobile provided with the driving environment adjustment device provided in the above embodiment. The device is particularly used for collecting images of drivers and passengers in the automobile by utilizing a camera which is arranged in the automobile at multiple angles; processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers; processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of a driver; and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters. Compared with the situation that environment adjustment can be carried out only based on the request of the user in the current intelligent cabin, the passenger behavior monitoring system can automatically monitor the passenger behavior of the driver and the passenger and adjust the environment, so that good care is automatically provided for the driver and the passenger, and the driving experience of the passenger is improved.
Example four
Fig. 8 is a block diagram of a controller of an automobile according to an embodiment of the present application.
The present embodiment provides an automobile provided with a controller, as shown in fig. 8, the controller includes at least one processor 101, and further includes a memory 102, and the memory is connected with the processor through a data bus 103.
The memory is used for storing a computer program or instructions, and the processor is used for executing the computer program or instructions, and by executing the program or instructions, the controller can be enabled to realize the riding environment adjusting method provided by the embodiment. The method specifically comprises the steps that a camera arranged in a car in multiple angles is used for collecting images of drivers and passengers in the car; processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers; processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of a driver; and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters. Compared with the situation that environment adjustment can be carried out only based on the request of the user in the current intelligent cabin, the passenger behavior monitoring system can automatically monitor the passenger behavior of the driver and the passenger and adjust the environment, so that good care is automatically provided for the driver and the passenger, and the driving experience of the passenger is improved.
EXAMPLE five
The present embodiment provides a storage medium having stored thereon program code adapted to be executed by a processor, the program code being configured to implement the following:
acquiring images of drivers and passengers in the automobile by utilizing the cameras arranged in the automobile at multiple angles;
processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers;
processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of the driver and passengers;
and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters.
The image includes a visible light image and/or an infrared light image.
The processing of the image using the bone detection model includes the steps of:
processing the image by utilizing a skeleton detection model based on an HRNet network;
or processing the image by utilizing a skeleton detection model based on an OpenPose network.
The step of adjusting the driving environment of the automobile according to the riding behavior and the environmental parameters comprises the following steps:
processing the riding behaviors and the environmental parameters based on a care service decision tree to obtain adjustment parameters;
And adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
Further comprising the steps of:
and carrying out model training based on a Baum _ Welch algorithm to obtain the passenger behavior recognition model.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The technical solutions provided by the present invention are described in detail above, and the principle and the implementation of the present invention are explained in this document by applying specific examples, and the descriptions of the above examples are only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (11)
1. A driving environment adjusting method is applied to an automobile, and is characterized by comprising the following steps:
acquiring images of drivers and passengers in the automobile by utilizing the cameras arranged in the automobile at multiple angles;
processing the image by using a skeleton detection model, and extracting a plurality of skeleton key points of the driver and passengers;
processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of the driver and passengers;
and adjusting the driving environment of the automobile according to the passenger behaviors and the environmental parameters.
2. The riding environment adjustment method of claim 1, wherein the image includes a visible light image and/or an infrared light image.
3. The method for regulating driving environment according to claim 1, wherein said processing said image using a skeleton detection model comprises the steps of:
processing the image by utilizing a skeleton detection model based on an HRNet network;
or processing the image by utilizing a skeleton detection model based on an OpenPose network.
4. The riding environment adjusting method according to claim 1, wherein the adjusting of the riding environment of the automobile according to the riding behavior and environmental parameters comprises the steps of:
Processing the riding behaviors and the environmental parameters based on a care service decision tree to obtain adjustment parameters;
and adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
5. The riding environment adjusting method according to any one of claims 1 to 4, further comprising the steps of:
and carrying out model training based on a Baum _ Welch algorithm to obtain the passenger behavior recognition model.
6. A riding environment adjusting apparatus applied to an automobile, the riding environment adjusting apparatus comprising:
the image acquisition module is used for receiving images of drivers and passengers in the automobile acquired by the cameras arranged in the automobile at multiple angles;
the first processing module is used for processing the image by utilizing a skeleton detection model and extracting a plurality of skeleton key points of the driver and passengers;
the second processing module is used for processing the plurality of bone key points by utilizing a passenger behavior recognition model based on a hidden Markov algorithm to obtain the passenger behavior of the driver;
and the adjusting execution module is used for adjusting the driving environment of the automobile according to the passenger behavior and the environmental parameters.
7. The riding environment adjustment device of claim 6, wherein the image includes a visible light image and/or an infrared light image.
8. The riding environment adjustment apparatus of claim 6, wherein the first processing module includes:
the first processing unit is used for processing the image by utilizing a skeleton detection model based on an HRnet network;
or the second processing unit is used for processing the image by utilizing a skeleton detection model based on an OpenPose network.
9. The riding environment adjustment device according to claim 6, wherein the adjustment execution module includes:
the environment calculation unit is used for processing the riding behaviors and the environment parameters based on a care service decision tree to obtain adjustment parameters;
and the parameter adjusting unit is used for adjusting part or all of the temperature, the window opening and the air volume of the automobile according to the adjusting parameters.
10. The riding environment adjustment device according to any one of claims 6 to 9, further comprising:
and the model training module is used for carrying out model training based on a Baum _ Welch algorithm to obtain the passenger behavior recognition model.
11. A storage medium characterized in that the storage medium has stored thereon program code that, when executed, realizes each step of the riding environment adjusting method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010697463.3A CN111860312A (en) | 2020-07-20 | 2020-07-20 | Driving environment adjusting method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010697463.3A CN111860312A (en) | 2020-07-20 | 2020-07-20 | Driving environment adjusting method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111860312A true CN111860312A (en) | 2020-10-30 |
Family
ID=73001294
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010697463.3A Pending CN111860312A (en) | 2020-07-20 | 2020-07-20 | Driving environment adjusting method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111860312A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112297768A (en) * | 2020-10-31 | 2021-02-02 | 中环曼普科技(南京)有限公司 | Vehicle air conditioner control method and system based on visual identification |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104038738A (en) * | 2014-06-04 | 2014-09-10 | 东北大学 | Intelligent monitoring system and intelligent monitoring method for extracting coordinates of human body joint |
| KR20150067679A (en) * | 2013-12-10 | 2015-06-18 | 현대자동차주식회사 | System and method for gesture recognition of vehicle |
| KR20180003134A (en) * | 2016-06-30 | 2018-01-09 | 주식회사 유투인터랙티브 | Apparatus and method for controlling indoor environment of vehicle using condition information of driver |
| CN108446678A (en) * | 2018-05-07 | 2018-08-24 | 同济大学 | A kind of dangerous driving behavior recognition methods based on skeleton character |
| CN110688929A (en) * | 2019-09-20 | 2020-01-14 | 北京华捷艾米科技有限公司 | Human skeleton joint point positioning method and device |
| CN110941990A (en) * | 2019-10-22 | 2020-03-31 | 泰康保险集团股份有限公司 | Method and device for evaluating human body actions based on skeleton key points |
| WO2020107847A1 (en) * | 2018-11-28 | 2020-06-04 | 平安科技（深圳）有限公司 | Bone point-based fall detection method and fall detection device therefor |
-
2020
- 2020-07-20 CN CN202010697463.3A patent/CN111860312A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20150067679A (en) * | 2013-12-10 | 2015-06-18 | 현대자동차주식회사 | System and method for gesture recognition of vehicle |
| CN104038738A (en) * | 2014-06-04 | 2014-09-10 | 东北大学 | Intelligent monitoring system and intelligent monitoring method for extracting coordinates of human body joint |
| KR20180003134A (en) * | 2016-06-30 | 2018-01-09 | 주식회사 유투인터랙티브 | Apparatus and method for controlling indoor environment of vehicle using condition information of driver |
| CN108446678A (en) * | 2018-05-07 | 2018-08-24 | 同济大学 | A kind of dangerous driving behavior recognition methods based on skeleton character |
| WO2020107847A1 (en) * | 2018-11-28 | 2020-06-04 | 平安科技（深圳）有限公司 | Bone point-based fall detection method and fall detection device therefor |
| CN110688929A (en) * | 2019-09-20 | 2020-01-14 | 北京华捷艾米科技有限公司 | Human skeleton joint point positioning method and device |
| CN110941990A (en) * | 2019-10-22 | 2020-03-31 | 泰康保险集团股份有限公司 | Method and device for evaluating human body actions based on skeleton key points |
Non-Patent Citations (1)
| Title |
|---|
| 毛峡;王岚;李建军;: "一种基于RGB-D特征融合的人体行为识别框架", 计算机科学, no. 08, 15 August 2018 (2018-08-15) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112297768A (en) * | 2020-10-31 | 2021-02-02 | 中环曼普科技(南京)有限公司 | Vehicle air conditioner control method and system based on visual identification |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106599773B (en) | Deep learning image identification method and system for intelligent driving and terminal equipment | |
| CN111267863B (en) | Driver driving type identification method and device, storage medium and terminal equipment | |
| KR102288280B1 (en) | Device and method to generate image using image learning model | |
| CN107194371B (en) | User concentration degree identification method and system based on hierarchical convolutional neural network | |
| CN111639540B (en) | Semi-supervised character re-recognition method based on camera style and human body posture adaptation | |
| CN108230291B (en) | Object recognition system training method, object recognition method, device and electronic equipment | |
| CN111258433B (en) | Teaching interaction system based on virtual scene | |
| CN107886043B (en) | Vision-aware anti-collision early warning system and method for forward-looking vehicles and pedestrians of automobile | |
| KR20050092230A (en) | Device for providing sound effect accrding to image and method thereof | |
| CN111696110B (en) | Scene segmentation method and system | |
| CN108009463B (en) | Identity recognition method and device | |
| CN116071817A (en) | Network architecture and training method of gesture recognition system for automobile cabin | |
| CN114037053A (en) | Vehicle visual perception data enhancement method based on GAN and related equipment | |
| CN111860312A (en) | Driving environment adjusting method and device | |
| KR101900185B1 (en) | Method and Device for Learning Image for Object Recognition | |
| DE102018002521A1 (en) | Method for recognizing persons | |
| CN110876011B (en) | Driving shooting method based on image recognition technology and vehicle | |
| CN113496472A (en) | Image defogging model construction method, road image defogging device and vehicle | |
| CN112560712B (en) | Behavior recognition method, device and medium based on time enhancement graph convolutional network | |
| CN106650754B (en) | A kind of sparse Gaussian process homing method for vision mapping | |
| CN112818887A (en) | Human body skeleton sequence behavior identification method based on unsupervised learning | |
| KR102240237B1 (en) | Apparatus and method for facial reproduction using genetic information | |
| CN108791308B (en) | System for constructing driving strategy based on driving environment | |
| CN114287008A (en) | Method for compressing neural networks | |
| CN117035125B (en) | Decision system, method and device of distributed artificial intelligence model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |