WO2024034920A1

WO2024034920A1 - Passenger detecting method, passenger detecting device, and passenger detecting system

Info

Publication number: WO2024034920A1
Application number: PCT/KR2023/010749
Authority: WO
Inventors: 유재윤; 강창범; 이현재
Original assignee: 주식회사 에너자이
Priority date: 2022-08-09
Filing date: 2023-07-25
Publication date: 2024-02-15
Also published as: KR102582729B1

Abstract

A passenger detecting method according to an embodiment of the present disclosure comprises the steps of: acquiring sensor data resulting from an electromagnetic wave which has been emitted through a radar sensor mounted inside a vehicle, and which is reflected by an object inside the vehicle; acquiring class information related to the status of a seat of the vehicle from the sensor data through a classification model which has finished learning; sensing a biometric signal by analyzing the sensor data; sensing a movement by analyzing the sensor data; and determining whether a passenger is aboard on the basis of the class information, the biometric signal sensing result, and the movement sensing result. The step of acquiring class information related to the status of a seat of the vehicle comprises the steps of: acquiring first initial class information of a first position of the seat of the vehicle through the classification model; acquiring second initial class information of a second position of the seat of the vehicle through the classification model; and operating the class information on the basis of the first initial class information and the second initial class information by using a tree-based classification algorithm.

Description

Passenger detection method, passenger detection device, and passenger detection system

This disclosure relates to passenger detection methods, passenger detection devices, and passenger detection systems. Specifically, the present disclosure relates to a passenger detection method, a passenger detection device, and a passenger detection system that can be executed on an MCU to be mounted in a vehicle.

Cameras have generally been widely used in human detection technology. However, there were limitations in using cameras to detect passengers inside a vehicle in that privacy issues may arise. Therefore, radar or ultrasonic waves have been used as alternative technologies to detect passengers inside a vehicle. However, conventional technologies for detecting passengers inside a vehicle using radar or ultrasonic waves were entirely rule-based, resulting in significant false alarms.

Meanwhile, as artificial intelligence technology develops, it is being applied to various industrial fields. However, given that the processor commonly installed in embedded devices such as vehicles is a microcontroller unit with relatively low specifications and low performance, there were limitations in optimally executing advanced algorithms such as deep learning in embedded devices. .

Accordingly, there is a need for the development of passenger detection methods, devices, and systems that can be executed on low-end processors while increasing the accuracy of passenger detection using artificial intelligence technology.

The problem that the present disclosure aims to solve is to provide a passenger detection method, a passenger detection device, and a passenger detection system to be executed on an MCU to be mounted in a vehicle.

The problem to be solved by the present disclosure is not limited to the above-mentioned problems, and problems not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. .

A passenger detection method according to an embodiment of the present disclosure includes the steps of acquiring sensor data in which electromagnetic waves transmitted through a radar sensor mounted inside a vehicle are reflected by an object inside the vehicle; Obtaining class information related to the status of the seat of the vehicle from the sensor data through a classification model on which learning has been completed - the class information indicates that the seat of the vehicle is empty. Class, a second class meaning that an adult is riding in the seat of the vehicle, and a third class meaning that a baby is riding in the seat of the vehicle; detecting biological signals by analyzing the sensor data; detecting movement by analyzing the sensor data; and determining whether or not a passenger is boarding based on the class information, the detection result of the biological signal, and the detection result of the movement. The step of obtaining class information related to the status of the seat of the vehicle includes, Obtaining first initial class information of a first position of a seat of the vehicle through the classification model; Obtaining second initial class information of a second position of the seat of the vehicle through the classification model; and calculating the class information based on the first initial class information and the second initial class information using a tree-based classification algorithm.

A passenger detection device according to an embodiment of the present disclosure includes a transceiver that receives sensor data in which electromagnetic waves transmitted through a radar sensor mounted inside the vehicle are reflected by an object inside the vehicle; and a processor configured to analyze the sensor data to determine whether a passenger has boarded the vehicle, wherein the processor acquires the sensor data and analyzes the sensor data through a fully learned classification model. Class information related to the status of the seat of the vehicle from data - the class information is a first class meaning that the seat of the vehicle is empty, and a second class meaning that an adult is riding in the seat of the vehicle. , and is related to a probability value corresponding to at least one of the third class, which means that a baby is riding in the seat of the vehicle, and detects a biological signal by analyzing the sensor data, and analyzes the sensor data It may be configured to detect movement and determine whether a passenger is boarding based on the class information, a detection result of a biological signal, and a detection result of the movement, and the processor is configured to determine whether a passenger is boarding a seat of the vehicle through the classification model. Obtain first initial class information of a position, obtain second initial class information of a second position of a seat of the vehicle through the classification model, and use a tree-based classification algorithm to obtain the first initial class information and the first position. 2 It may be configured to obtain class information related to the state of the seat of the vehicle by calculating the class information based on initial class information.

The technical solutions of the present disclosure are not limited to the above-described technical solutions, and technical solutions not mentioned are clearly understood by those skilled in the art from this specification and the attached drawings. It could be.

According to the passenger detection method, passenger detection device, and passenger detection system according to an embodiment of the present disclosure, the size of the classification model can be relatively lightweight, thereby providing the effect of being able to run efficiently even in an MCU environment mounted in a vehicle.

According to the passenger detection method, passenger detection device, and passenger detection system according to an embodiment of the present disclosure, the accuracy of analysis related to the state of seats in the vehicle can be increased through a tree-based classification algorithm.

According to the passenger detection method, passenger detection device, and passenger detection system according to an embodiment of the present disclosure, when determining whether to board a passenger, whether or not a biological signal is detected and whether a motion signal is detected is additionally considered to board the passenger. It can provide the effect of increasing the accuracy and reliability of judgment.

The effects of the present disclosure are not limited to the effects described above, and effects not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings.

1 is a schematic diagram of a passenger detection system according to an embodiment of the present disclosure.

Figure 2 is a diagram showing the operation of a passenger detection device according to an embodiment of the present disclosure.

Figure 3 is a flowchart showing a passenger detection method according to an embodiment of the present disclosure.

Figure 4 is a flow chart specifying steps for acquiring sensor data according to an embodiment of the present disclosure.

FIG. 5 is a diagram illustrating an aspect of acquiring class information through a fully trained classification model according to an embodiment of the present disclosure.

FIG. 6 is a diagram illustrating an aspect of learning a classification model according to an embodiment of the present disclosure.

Figure 7 is a flow chart specifying the steps of obtaining class information according to an embodiment of the present disclosure.

Figure 8 is a flow chart specifying steps for calculating class information according to an embodiment of the present disclosure.

FIG. 9 is a diagram for explaining an aspect of calculating class information according to an embodiment of the present disclosure.

Figure 10 is a flowchart detailing the steps of detecting a biological signal according to an embodiment of the present disclosure.

Figure 11 is a flowchart detailing steps for detecting movement according to an embodiment of the present disclosure.

Figure 12 is a diagram for explaining aspects of determining whether a passenger is boarding according to an embodiment of the present disclosure.

The above-described objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, since the present disclosure can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail below.

Like reference numerals throughout the specification in principle refer to the same elements. In addition, components with the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present disclosure may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of this specification are merely identifiers to distinguish one component from another component.

In addition, the suffixes “module” and “part” for components used in the following examples are given or used interchangeably only considering the ease of writing the specification, and do not have distinct meanings or roles in themselves.

In the following examples, singular terms include plural terms unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean that the features or components described in the specification exist, and do not exclude in advance the possibility of adding one or more other features or components.

In the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present disclosure is not necessarily limited to what is shown.

If an embodiment can be implemented differently, the order of specific processes may be performed differently from the order described. For example, two processes described in succession may be performed substantially simultaneously, or may proceed in an order opposite to that in which they are described.

In the following embodiments, when components are connected, this includes not only the case where the components are directly connected, but also the case where the components are indirectly connected by intervening between the components.

For example, in this specification, when components, etc. are said to be electrically connected, this includes not only cases where the components are directly electrically connected, but also cases where components, etc. are interposed and indirectly electrically connected.

A passenger detection method according to an embodiment of the present disclosure includes obtaining sensor data in which electromagnetic waves transmitted through a radar sensor mounted inside a vehicle are reflected by an object inside the vehicle; Obtaining class information related to the status of the seat of the vehicle from the sensor data through a classification model on which learning has been completed - the class information indicates that the seat of the vehicle is empty Class, a second class meaning that an adult is riding in the seat of the vehicle, and a third class meaning that a baby is riding in the seat of the vehicle; detecting biological signals by analyzing the sensor data; detecting movement by analyzing the sensor data; and determining whether to board a passenger based on the class information, biometric signal detection results, and movement detection results.

According to an embodiment of the present disclosure, the step of determining whether a passenger is on board includes determining that a passenger is on board the vehicle when the class information is obtained as the second class or the third class and the biosignal is detected. and, if the class information is obtained as the second class or the third class and the biosignal is not detected, it is determined that a passenger is not aboard the vehicle, and if the movement is detected, the passenger is aboard the vehicle. It may further include determining that movement of an object, including a person, has been detected inside the vehicle, regardless of whether the motion is detected.

According to an embodiment of the present disclosure, the learned classification model receives the sensor data and selects at least one of the first class, the second class, and the third class of the class information based on the sensor data. It may be configured to output a probability value corresponding to one.

According to an embodiment of the present disclosure, the trained classification model includes an input layer that receives learning data consisting of sensor data and labels assigned to the sensor data, an output layer that outputs a predicted value, and the input layer and the output It includes a hidden layer connecting layers, and can be trained by adjusting the weight of at least one node constituting the hidden layer based on the difference between the predicted value and the label.

According to an embodiment of the present disclosure, the step of acquiring class information related to the status of the seat of the vehicle includes obtaining first initial class information of the first position of the seat of the vehicle through the classification model. step; Obtaining second initial class information of a second position of the seat of the vehicle through the classification model; and calculating the class information based on the first initial class information and the second initial class information using a tree-based classification algorithm.

According to an embodiment of the present disclosure, the step of calculating the class information includes the first initial class information at the first location and the first rule node through the first rule node included in the tree-based classification algorithm. comparing first conditions; Comparing second initial class information at the second location and a second condition of the second rule node through a second rule node included in the tree-based classification algorithm and connected to the first rule node; and determining class information related to the state of the seat of the vehicle based on the comparison result.

According to an embodiment of the present disclosure, acquiring the sensor data includes acquiring raw data reflected by an object inside the vehicle; performing a Fast Fourier Transform on the raw data in a first direction of a spherical coordinate system to transform the raw data and obtain transformed data; and performing capon beamforming on the converted data to obtain a first heat map related to a second direction of a spherical coordinate system and a second heat map related to a third direction of a spherical coordinate system to obtain the sensor data. May include ;.

According to an embodiment of the present disclosure, the step of detecting the biological signal includes accumulating the conversion data by a predetermined time window and obtaining accumulated data; determining whether the accumulated data satisfies a predetermined rule; and determining whether a biological signal is detected in a seat inside the vehicle based on the determination result.

According to one embodiment of the present disclosure, detecting the motion may include calculating an average value of values corresponding to the intensity of a signal of at least one of the first heat map and the second heat map; obtaining a predetermined threshold; and comparing the average value with the threshold value, and determining that movement has been detected inside the vehicle if the average value is greater than or equal to the threshold value.

According to an embodiment of the present disclosure, a computer-readable recording medium recording a program for executing the passenger detection method may be provided.

A passenger detection device according to an embodiment of the present disclosure includes a transceiver that receives sensor data in which electromagnetic waves transmitted through a radar sensor mounted inside the vehicle are reflected by an object inside the vehicle; and a processor configured to analyze the sensor data to determine whether a passenger has boarded the vehicle, wherein the processor acquires the sensor data and analyzes the sensor data through a fully learned classification model. Class information related to the status of the seat of the vehicle from data - the class information is a first class meaning that the seat of the vehicle is empty, and a second class meaning that an adult is riding in the seat of the vehicle. , and is related to a probability value corresponding to at least one of the third class, which means that a baby is riding in the seat of the vehicle, and detects a biological signal by analyzing the sensor data, and analyzes the sensor data It may be configured to detect movement and determine whether to board a passenger based on the class information, the detection result of biometric signals, and the detection result of the movement.

Hereinafter, the passenger detection method, passenger detection device, and passenger detection system of the present disclosure will be described with reference to FIGS. 1 to 12.

The passenger detection system 10 according to an embodiment of the present disclosure may include a radar sensor 100 and a passenger detection device 1000 (or passenger detection server, hereinafter referred to as a passenger detection device).

The radar sensor 100 according to an embodiment of the present disclosure may include a transmitter and a receiver. Specifically, the radar sensor 100 can transmit electromagnetic waves in a specific frequency range through a transmitter. Additionally, the radar sensor 100 may receive sensor data from a signal reflected from an arbitrary object through a receiver. Additionally, the radar sensor 100 may calculate location information and speed information of an object based on the received sensor data. For example, the radar sensor 100 may be a radar sensor that transmits and receives a frequency modulated continuous wave (FMCW). However, this is only an example, and a radar that transmits and receives any type of signal can be used as the radar sensor 100.

The radar sensor 100 according to an embodiment of the present disclosure may be placed at any location inside the vehicle. For example, the radar sensor 100 may be placed on the upper side of the vehicle interior. However, this is just an example and may be placed in any appropriate location.

The passenger detection device 1000 according to an embodiment of the present disclosure may acquire sensor data from the radar sensor 100 and analyze the sensor data to determine whether a passenger is on board. As an example, the passenger detection device 1000 may analyze sensor data to obtain class information related to the status of seats (eg, rear seat, driver's seat, passenger seat) of the vehicle. For example, the passenger detection device 1000 may obtain class information related to the status of the vehicle's seats (eg, rear seat, driver's seat, passenger seat) from sensor data through a fully learned classification model. As another example, the passenger detection device 1000 may detect biosignals by analyzing sensor data. Specifically, the passenger detection device 1000 may analyze sensor data to determine whether a biological signal is detected inside the vehicle. As another example, the passenger detection device 1000 may detect movement inside the vehicle by analyzing sensor data. Specifically, the passenger detection device 1000 determines whether the value corresponding to the strength of the signal included in the sensor data is greater than or equal to a predetermined threshold, and determines whether movement inside the vehicle has been detected based on the determination result. You can judge.

Below, we will focus on detecting passengers in the rear seat of a vehicle. However, this is only for convenience of explanation and should not be interpreted as limiting, and may be similarly applied to detecting passengers in the driver's seat and/or passenger seat of a vehicle.

The passenger detection device 1000 according to an embodiment of the present disclosure may include a transceiver 1100, a memory 1200, and a processor 1300.

The transceiver unit 1100 of the passenger detection device 1000 may communicate with any external device, including the radar sensor 100. For example, the passenger detection device 1000 may receive sensor data acquired through the radar sensor 100 through the transceiver 1100. Additionally, the passenger detection device 1000 may obtain execution data for executing a classification model on which learning has been completed through the transceiver 1100. Here, execution data may mean encompassing arbitrary data for properly executing the learned classification model, including layer information, operation information, and weight information of the learned classification model. Additionally, the passenger detection device 1000 may transmit passenger detection information analyzed from sensor data to any external device or external server through the transceiver 1100.

The passenger detection device 1000 can connect to a network and transmit and receive various data through the transmitting and receiving unit 1100. Transmitter/receiver units may broadly include wired types and wireless types. Since the wired type and the wireless type have their own advantages and disadvantages, in some cases, the passenger detection device 1000 may be provided with both the wired type and the wireless type. Here, in the case of the wireless type, a WLAN (Wireless Local Area Network) type communication method such as Wi-Fi can be mainly used. Alternatively, in the case of the wireless type, cellular communication, such as LTE or 5G communication methods, can be used. However, the wireless communication protocol is not limited to the above-described example, and any appropriate wireless type of communication method can be used. In the case of the wired type, LAN (Local Area Network) or USB (Universal Serial Bus) communication are representative examples, but other methods are also possible.

The memory 1200 of the passenger detection device 1000 can store various types of information. Various data may be temporarily or semi-permanently stored in the memory 1200. Examples of memory may include hard disk drives (HDD), solid state drives (SSD), flash memory, read-only memory (ROM), and random access memory (RAM). there is. The memory 1200 may be provided in a form built into the passenger detection device 1000 or in a detachable form. The memory 1200 includes an operating program (OS: Operating System) for operating the passenger detection device 1000 or a program for operating each component of the passenger detection device 1000, as well as information necessary for the operation of the passenger detection device 1000. Various data can be stored.

The processor 1300 may control the overall operation of the passenger detection device 1000. For example, the processor 1300 may perform an operation of acquiring sensor data, which will be described later, an operation of acquiring class information related to the state of the rear seat of the vehicle through a fully learned classification model, an operation of detecting a biological signal by analyzing sensor data, and sensor data. It is possible to control the overall operation of the passenger detection device 1000, including an operation of detecting movement by analyzing and/or an operation of determining whether a passenger is on board. Specifically, the processor 1300 may load and execute a program for the overall operation of the passenger detection device 1000 from the memory 1200. The processor 1300 may be implemented as an application processor (AP), a central processing unit (CPU), a microcontroller unit (MCU), or a similar device depending on hardware, software, or a combination thereof. At this time, hardware may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and software may be provided in the form of a program or code that drives the hardware circuit.

Hereinafter, with reference to FIGS. 2 to 12, the operation of the passenger detection device 1000 and the passenger detection method according to an embodiment of the present disclosure will be described in detail.

Figure 2 is a diagram showing the operation of the passenger detection device 1000 according to an embodiment of the present disclosure.

The passenger detection device 1000 according to an embodiment of the present disclosure may acquire sensor data from the radar sensor 100 through the transceiver 1100. Sensor data used throughout the specification includes raw data of sensor data reflected on an object, and/or raw data processed data, and arbitrary data derived from sensor data received through the radar sensor 100. It is meant to encompass data in the form of . In other words, sensor data can be used to encompass raw data, converted data, and/or heatmap data, which will be described later, and should not be interpreted as limited due to the mixed use of expressions.

Meanwhile, the passenger detection device 1000 according to an embodiment of the present disclosure may perform an operation of performing pre-processing on raw data received from the radar sensor 100. As an example, the passenger detection device 1000 may perform Fast Fourier Transform on raw data. As another example, the passenger detection device 1000 may perform an operation to remove a specific signal included in raw data. For example, the passenger detection device 1000 may perform clutter removal to remove noise signals (eg, signals corresponding to non-moving objects) included in raw data. As another example, the passenger detection device 1000 may perform an operation to obtain heat map data from sensor data using a beamforming technique. For example, the passenger detection device 1000 may perform car phone beamforming to obtain a heat map from sensor data (or raw data). The operation of the passenger detection device 1000 that acquires sensor data will be described in more detail later with reference to FIGS. 3 and 4 .

The passenger detection device 1000 according to an embodiment of the present disclosure may acquire class information related to the state of the rear seat of the vehicle through a fully learned classification model. Specifically, the passenger detection device 1000 may input sensor data into a fully learned classification model and obtain class information output through the fully learned classification model. Here, the class information includes the first class meaning that the rear seat of the vehicle is empty, the second class meaning that an adult is riding in the rear seat of the vehicle, and the first class meaning that a baby is riding in the rear seat of the vehicle. It may be related to a probability value corresponding to at least one of the three classes. A classification model may be learned based on training data consisting of sensor data and labels assigned to the sensor data. Specifically, the classification model may be configured to output a predicted value based on learning data consisting of sensor data and labels assigned class information of the sensor data. At this time, the classification model may be learned by updating at least one parameter included in the classification model based on the difference between the predicted value and the label.

Meanwhile, the passenger detection device 1000 according to an embodiment of the present disclosure may be implemented to perform a post-processing operation on class information obtained through a classification model. As an example, the passenger detection device 1000 may be implemented to correct class information obtained through a classification model using a tree-based classification algorithm.

The operation of the passenger detection device 1000 that acquires class information will be described in more detail later with reference to FIGS. 5 to 9.

The passenger detection device 1000 according to an embodiment of the present disclosure may perform an operation of detecting biological signals by analyzing sensor data. Specifically, the passenger detection device 1000 may determine whether sensor data satisfies a predetermined rule, and determine whether a biological signal has been detected from the rear seat inside the vehicle based on the determination result. For example, if the sensor data satisfies a predetermined rule, the passenger detection device 1000 may determine that a biosignal has been detected from the rear seat of the vehicle, and if the sensor data does not satisfy the predetermined rule, the passenger detection device 1000 may determine that a biosignal has been detected from the rear seat of the vehicle. It can be determined that biological signals were not detected.

The operation of the passenger detection device 1000, which detects biological signals by analyzing sensor data, will be described in more detail later with reference to FIG. 10.

The passenger detection device 1000 according to an embodiment of the present disclosure may perform an operation to detect movement inside the vehicle by analyzing sensor data. Specifically, the passenger detection device 1000 may determine whether there is movement inside the vehicle based on a value corresponding to the strength of the signal of sensor data. For example, the passenger detection device 1000 calculates the average value of values corresponding to the signal strength of the heat map, and when the calculated average value is greater than or equal to a predetermined threshold, it determines that movement has been detected inside the vehicle. It can be implemented. On the other hand, the passenger detection device 1000 may be implemented to determine that no movement has been detected inside the vehicle when the calculated average value is less than a predetermined threshold. Here, 'movement' means not only the movement of people, but also the movement of any object that occurs inside the vehicle.

The operation of the passenger detection device 1000, which detects movement by analyzing sensor data, will be described in more detail later with reference to FIG. 11.

The passenger detection device 1000 according to an embodiment of the present disclosure can determine whether a passenger is riding inside the vehicle (eg, rear seat). Specifically, the passenger detection device 1000 may be implemented to determine whether to board a passenger based on class information, whether biometric signals are detected, and whether movement is detected. As an example, if class information is obtained as one of the second class, meaning that an adult is riding in the rear seat of the vehicle, or the third class, meaning that a baby is riding in the rear seat of the vehicle, and a biological signal is detected, a passenger is detected. The device 1000 may be implemented to determine that a passenger is boarding the vehicle. As another example, if the class information is obtained as one of the second class, meaning that an adult is riding in the rear seat of the vehicle, or the third class, meaning that a baby is riding in the rear seat of the vehicle, and no vital signals are detected, The passenger detection device 1000 may be implemented to determine that no passengers are boarding the vehicle. As another example, when movement is detected, the passenger detection device 1000 may be implemented to determine that movement of an object, including a person, has been detected inside the vehicle regardless of whether a passenger is in the vehicle. The operation of the passenger detection device 1000 that determines whether a passenger is on board will be described in more detail with reference to FIG. 12.

Hereinafter, with reference to FIGS. 3 to 12, a passenger detection method performed by the passenger detection device 1000 according to an embodiment of the present disclosure will be described in detail. In describing the passenger detection method, some embodiments that overlap with the content previously described in relation to FIG. 2 may be omitted. However, this is only for convenience of explanation and should not be construed as limiting.

The passenger detection method according to an embodiment of the present disclosure includes acquiring sensor data (S1000), using a learned classification model to obtain class information related to the status of the rear seat of the vehicle from sensor data. It may include a step (S2000), a step of analyzing sensor data to detect biosignals (S3000), a step of analyzing sensor data to detect movement (S4000), and a step of determining whether a passenger is on board (S5000). .

In the step of acquiring sensor data (S1000), the passenger detection device 1000 may acquire sensor data from the radar sensor 100 through the transceiver 1100. Here, as described above, the sensor data originates from sensor data received through the radar sensor 100, including raw data of sensor data reflected on an object, and/or raw data processed data. It is meant to encompass data in any form.

Additionally, in the step of acquiring sensor data (S1000), the passenger detection device 1000 may perform pre-processing on the raw data received from the radar sensor 100. Hereinafter, with reference to FIG. 4, obtaining sensor data by performing preprocessing on raw data will be described in more detail.

Figure 4 is a flow chart specifying the step (S1000) of acquiring sensor data according to an embodiment of the present disclosure.

The step of acquiring sensor data (S1000) according to an embodiment of the present disclosure includes the step of acquiring raw data (S1100), by performing Fast Fourier Transform on the raw data in the first direction of the spherical coordinate system. Converting raw data and obtaining converted data (S1200), and performing Capon Beamforming on the converted data to create a first heatmap related to the second direction of the spherical coordinate system and a third heatmap of the spherical coordinate system. It may further include acquiring sensor data by acquiring a second heat map related to the direction (S1300).

In the step of acquiring raw data (S1100), the passenger detection device 1000 may acquire raw data through the transceiver 1100. For example, raw data may be data in the form of frequency according to time.

In the step (S1200) of transforming the raw data and obtaining transformed data by performing Fast Fourier Transform on the raw data in the first direction of the spherical coordinate system, the passenger detection device 1000 performs Fast Fourier Transform on the raw data. Conversion can be performed. Specifically, the passenger detection device 1000 may perform fast Fourier transformation in the first direction (eg, range direction) of the spherical coordinate system to transform raw data and obtain transformed data. Through this, the passenger detection device 1000 can convert time domain data into frequency domain data.

Obtaining sensor data by performing Capon Beamforming on the converted data to obtain a first heatmap related to the second direction of the spherical coordinate system and a second heatmap related to the third direction of the spherical coordinate system In (S1300), the passenger detection device 1000 may perform heat map data from sensor data using a beamforming technique. Specifically, the passenger detection device 1000 may perform car phone beamforming on the converted data to obtain heat map data from the converted data.

As an example, the passenger detection device 1000 may perform car phone beamforming on the converted data to obtain a first heatmap related to the second direction (eg, theta direction) of the spherical coordinate system. Specifically, the passenger detection device 1000 decomposes the signal with respect to the second direction of the spherical coordinate system through a car phone beamforming technique while fixing the value for the third direction (e.g., phi direction) of the spherical coordinate system, and converts the data into converted data. The first heat map can be generated from.

As another example, the passenger detection device 1000 may perform car phone beamforming on the converted data to obtain a second heatmap related to the third direction (eg, pi direction) of the spherical coordinate system. Specifically, the passenger detection device 1000 decomposes the signal in the third direction of the spherical coordinate system through a car phone beamforming technique while fixing the value for the second direction (e.g., theta direction) of the spherical coordinate system, and converts the data into converted data. A second heat map can be generated from.

Meanwhile, Capon Beamforming is performed on the converted data to obtain a first heatmap related to the second direction of the spherical coordinate system and a second heatmap related to the third direction of the spherical coordinate system to obtain sensor data. In step S1300, the passenger detection device 1000 may acquire at least one first heat map and at least one second heat map. For example, the passenger detection device 1000 may acquire two or more first heat maps and two second heat maps, respectively. For example, the passenger detection device 1000 may acquire three or more first heat maps and three second heat maps, respectively. However, this is only an example, and the passenger detection device 1000 may be implemented to acquire an arbitrary appropriate number of heat maps in consideration of the amount of memory required for analysis of sensor data and acquisition of information on areas of interest inside the vehicle. You will be able to.

Meanwhile, although not shown in FIG. 4, in the step (S1000) of acquiring sensor data according to an embodiment of the present disclosure, the passenger detection device 1000 removes a specific signal included in the raw data (or converted data). You can. For example, the passenger detection device 1000 may use a clutter removal technique to remove noise signals (eg, signals corresponding to non-moving objects) included in raw data. More specifically, the passenger detection device 1000 may remove noise signals included in raw data by applying clutter removal to specific window units (eg, multiple frames). At this time, by performing clutter removal on a specific window unit, only signals for non-moving objects can be removed without attenuating biological signals (eg, respiration signals, heart rate signals, etc.).

In addition, although not shown in FIG. 4, in the step (S1000) of acquiring sensor data according to an embodiment of the present disclosure, the passenger detection device 1000 acquires sensor data corresponding to a plurality of frames, and the obtained sensor data Data can be merged. For example, the passenger detection device 1000 may acquire at least one first heatmap and a second heatmap obtained from the current frame, and at least one first heatmap and a second heatmap obtained from the previous frame. At this time, the passenger detection device 1000 may combine the obtained first heatmaps of the current frame and the first heatmaps of the previous frame. Additionally, the passenger detection apparatus 1000 may combine the acquired second heatmaps of the current frame and the second heatmaps of the previous frame. Through this, the passenger detection device 1000 can perform analysis on the sensor data by reflecting the time-series characteristics of the sensor data, thereby increasing the accuracy of the sensor data analysis.

In the step (S2000) of acquiring class information related to the status of the rear seat of the vehicle from sensor data using the learned classification model, the passenger detection device 1000 is used to execute the learned classification model. Execution data (eg, layer information, operation information, and/or weight information of a classification model) may be obtained. In addition, the passenger detection device 1000 executes the learned classification model using execution data, inputs sensor data into the learned classification model, and outputs the status of the rear seat of the vehicle through the learned classification model. You can obtain class information related to . Here, the class information includes the first class meaning that the rear seat of the vehicle is empty, the second class meaning that an adult is riding in the rear seat of the vehicle, and the first class meaning that a baby is riding in the rear seat of the vehicle. It may be related to a probability value corresponding to at least one of the three classes.

The trained classification model receives sensor data (e.g., a first heatmap and a second heatmap), and based on the sensor data, a first class indicating that the rear seat of the vehicle is empty, an adult in the rear seat of the vehicle, and a person in the rear seat of the vehicle. It may be configured to output class information related to a probability value corresponding to at least one of a second class, meaning that a baby is riding in the rear seat of the vehicle, and a third class, meaning that a baby is riding in the rear seat of the vehicle. Accordingly, the passenger detection device 1000 can be implemented to obtain class information from sensor data through a fully learned classification model.

Meanwhile, a classification model that has completed learning may be configured to have multiple branches. As an example, the trained classification model has a first branch and a second branch, and has a structure in which the first branch and the second branch are merged in a layer adjacent to the output layer (e.g., at least one convolution adjacent to the output layer) The solution layer can be configured to have a common structure. At this time, the passenger detection device 1000 inputs the first heatmap related to the second direction of the spherical coordinate system to the first branch, inputs the second heatmap related to the third direction of the spherical coordinate system to the second branch, and outputs It can be implemented to obtain class information output through the layer.

Furthermore, the learned classification model can separate the output layer according to the position of the rear seat of the vehicle. For example, if the positions of the rear seats of the vehicle are divided into five positions (e.g., 3 rows of rear seats, leg room, trunk), the output layer may be separated by the positions of the rear seats.

According to an embodiment of the present disclosure, the layer adjacent to the output layer has a common structure for the first heat map and the second heat map, thereby reducing the amount of memory required for calculation, and through this, computing specifications are relative. This can provide the advantage of being able to analyze sensor data using a classification model even in a low MCU environment.

A classification model may include an input layer, an output layer, and a hidden layer connecting the input layer and the output layer.

The input layer may be configured to receive training data consisting of sensor data and labels assigned to the sensor data. Additionally, the label may be a value corresponding to class information assigned to the sensor data, and the class information includes, as described above, the first class indicating that the rear seat of the vehicle is empty, and the first class indicating that an adult is riding in the rear seat of the vehicle. It may be related to a probability value corresponding to at least one of the second class, meaning that a baby is riding in the rear seat of the vehicle, and the third class, meaning that a baby is riding in the rear seat of the vehicle. Additionally, the classification model may be structured to have at least one input layer. For example, the classification model may have a structure consisting of a first input layer for receiving sensor data corresponding to the first heatmap and a second input layer for receiving sensor data corresponding to the second heatmap.

The output layer may be configured to output a predicted value calculated based on learning data. At this time, as described above, the classification model may have a structure in which the output layer is divided by the position of the rear seat of the vehicle. For example, the output layer includes a first output layer that outputs a predicted value corresponding to the left position of the rear seat, a second output layer that outputs a predicted value corresponding to the right position of the rear seat, and a third output layer that outputs a predicted value corresponding to the center position of the rear seat. It may be configured to be divided into an output layer, a fourth output layer that outputs a predicted value corresponding to the legroom position, and/or a fifth output layer that outputs a predicted value corresponding to the trunk position.

The hidden layer connects the input layer and the output layer and may include at least one node.

The passenger detection device (1000, or any external device) may be implemented to obtain the predicted value output through the output layer and compare the predicted value with the label included in the learning data. Additionally, the passenger detection device 1000 may update the weight (or parameter) of at least one node of the classification model based on the difference between the label and the predicted value. Specifically, the passenger detection device 1000 may update at least one weight of the hidden layer so that the difference between the predicted value and the label is minimized.

The passenger detection device 1000 (or any external device) according to an embodiment of the present disclosure may perform an operation to reduce the weight of the classification model.

As an example, a classification model may be trained as a sub-net that forms part of a super network. More specifically, a super network may be composed of a classification model constituting a subnet, a connection to the subnet, and a plugnet. At this time, the passenger detection device (1000 or any external device) can acquire a lightweight classification model by learning a super network based on the training data and extracting a subnet that forms part of the super network.

As another example, the passenger detection device (1000 or any external device) may perform integerization on the input data and/or output data of the operation constituting the classification model to make the classification model lightweight. Specifically, the passenger detection device 1000 may utilize a quantization technique to convert input data and/or output data of decimal-type operations into integers in a specific range. Here, quantization is a technique that converts a 32-bit decimal point (float) value into an 8-bit integer (int). The passenger detection device 1000 calculates the scale and zero point for each data tensor for the operation constituting the classification model, converting the 32-bit decimal value into an 8-bit integer value (int8Value) with a relatively small capacity. By converting to , the classification model can be made lightweight.

Meanwhile, the learned classification model can be executed in a microcontroller unit (MCU) environment to be installed in the vehicle. However, MCUs have relatively limited computing specifications and resources, so there is a risk that running a classification model to analyze sensor data in an MCU environment will act as a load on memory. Accordingly, the passenger detection device 1000 (or any external device) according to an embodiment of the present disclosure may perform an operation to optimize the classification model.

As an example, the passenger detection device (1000, or any external device) according to an embodiment of the present disclosure determines the execution order of the operations of the classification model based on the execution data of the classification model and corresponds to a predetermined reference operation pattern. The target operation pattern may be detected, and an operation of merging the first target operation and the second operation included in the target operation pattern may be performed. Specifically, the standard operation pattern relates to a commonly used operation pattern. For example, the reference operation pattern may be related to an operation pattern that sequentially performs a convolution operation, a depthwise convolution operation, and an activation operation. As another example, the reference operation pattern may be related to an operation pattern that performs a convolution operation and a Rectified Linear Unit (ReLu) operation. At this time, the passenger detection device 1000 may detect a target operation pattern that corresponds to the reference operation pattern and is included in the classification model, and perform an optimization operation to merge operations included in the target operation pattern.

As another example, the passenger detection device 1000 (or any external device) according to an embodiment of the present disclosure obtains a memory space map based on the execution order of operations of the classification model, and memory allocation and memory allocation based on the memory space map. Relevant optimizations can be performed. For example, the passenger detection device 1000 utilizes a memory in-placing technique to convert the output space of a specific operation (e.g., ReLu operation, Add operation, and/or Sigmoid operation, etc.) into the input space of the corresponding operation. You can perform an operation to overwrite space. Specifically, the passenger detection device 1000 will be implemented to change the memory tensor in which the value input by a specific operation of the classification model is stored into a memory tensor in which the value output through a specific operation is stored by utilizing the memory in-place technique. You can.

According to an embodiment of the present disclosure, the passenger detection device 1000 may determine whether a passenger is boarding based on an output value (i.e., class information) obtained through a fully learned classification model.

However, according to another embodiment of the present disclosure, the passenger detection device 1000 acquires class information (hereinafter, initial class information) through a learned classification model in order to minimize errors in determining whether or not to board a passenger. It can be implemented to perform a post-processing operation. According to one embodiment, the passenger detection device 1000 may perform post-processing on initial class information using a tree-based classification algorithm and obtain class information. Hereinafter, with reference to FIGS. 7 to 9, obtaining class information using a tree-based classification algorithm according to an embodiment of the present disclosure will be described in more detail.

Figure 7 is a flow chart specifying the step (S2000) of acquiring class information according to an embodiment of the present disclosure.

Obtaining class information according to an embodiment of the present disclosure (S2000) includes acquiring first initial class information of the first position of the rear seat of the vehicle through a classification model (S2100), and obtaining first initial class information of the first position of the rear seat of the vehicle through a classification model. A step of acquiring second initial class information at the second position of the analysis (S2200), and a step of calculating class information based on the first initial class information and the second initial class information using a tree-based classification algorithm (S2300). More may be included.

In the step (S2100) of acquiring the first initial class information of the first position of the rear seat of the vehicle through the classification model, the passenger detection device 1000 determines the rear seat of the vehicle output through the output layer of the classification model for which learning has been completed. First initial class information related to the state of the first location may be obtained. For example, the passenger detection device 1000 may obtain first initial class information corresponding to the left position of the rear seat of the vehicle output through the output layer of the classification model for which learning has been completed.

In the step (S2200) of acquiring the second initial class information of the second position of the rear seat of the vehicle through the classification model, the passenger detection device 1000 determines the rear seat of the vehicle output through the output layer of the classification model for which learning has been completed. Second initial class information related to the state of the second location may be obtained. For example, the passenger detection device 1000 may obtain second initial class information corresponding to the right position of the rear seat of the vehicle output through the output layer of the classification model for which learning has been completed.

In the step (S2300) of calculating class information based on the first initial class information and the second initial class information using a tree-based classification algorithm, the passenger detection device 1000 uses a tree-based classification algorithm. Class information can be calculated based on the first initial class information and the second initial class information using . A tree-based classification algorithm can be composed of a rule node containing preset rules (or setting values) and a leaf node (Terminal Node) where classification values are output or stored through the rule node. At this time, the passenger detection device 1000 is configured to obtain class information depending on whether the first initial class information and/or the second initial class information satisfies a predetermined rule of at least one rule node of the tree-based classification algorithm. It can be implemented.

Hereinafter, with reference to FIGS. 8 and 9, calculating class information using a tree-based classification algorithm according to an embodiment of the present disclosure will be described in more detail.

Figure 8 is a flow chart specifying the step (S2300) of calculating class information according to an embodiment of the present disclosure. FIG. 9 is a diagram for explaining an aspect of calculating class information according to an embodiment of the present disclosure.

The step of calculating class information (S2300) according to an embodiment of the present disclosure includes first initial class information at the first location and the first condition of the first rule node through the first rule node included in the tree-based classification algorithm. Comparing step (S2310), comparing the second initial class information at the second location and the second condition of the second rule node through the second rule node included in the tree-based classification algorithm and connected to the first rule node ( S2320), and determining class information related to the state of the rear seat of the vehicle based on the comparison result (S2330).

In the step (S2310) of comparing the first initial class information at the first location and the first condition of the first rule node through the first rule node included in the tree-based classification algorithm, the passenger detection device 1000 uses a tree-based classification algorithm. Obtain the first condition of the first rule node included in the classification algorithm, and compare the first condition with the first initial class information at the first position to determine whether the first initial class information satisfies the first condition. You can. Specifically, a tree-based classification algorithm includes a root node, which is the top node of the tree, at least one rule node, which is a node that generates rule conditions, and a leaf node, where the final classification value is determined, and It may be composed of a sub tree, which is a tree created for each new rule condition.

At this time, the passenger detection device 1000 may compare the first condition (or first setting value) of the first rule node included in the subtree of the tree-based classification algorithm with the first initial class information. For example, the passenger detection device 1000 may include first initial class information (e.g., a probability value that a passenger (e.g., adult, baby) is riding in the first location) related to the first location (e.g., the right side position of the rear seat) and 1 It can be compared with the first condition of the rule node, and it can be determined whether the first initial class information satisfies the first condition. For example, the first condition may be a condition that the first initial class information is greater than, equal to, or less than the first threshold probability value, and in this case, the first initial class information (e.g., in the form of a probability value) does not satisfy the first condition. If not, the passenger detection device 1000 may determine a classification value related to class information through the first leaf node below the first rule node. On the other hand, if the first initial class information (e.g., in the form of a probability value) satisfies the second condition, additional classification may be performed through a second rule node below the first rule node.

In the step (S2320) of comparing the second initial class information at the second location and the second condition of the second rule node through the second rule node included in the tree-based classification algorithm and connected to the first rule node, the passenger detection device ( 1000), the passenger detection device 1000 obtains the second condition of the second rule node included in the tree-based classification algorithm, compares the second initial class information at the second location and the second condition, and determines the second initial class information. It can be determined whether the class information satisfies the second condition. Specifically, the passenger detection device 1000 may compare the second condition (or second setting value) of the second rule node of the tree-based classification algorithm with the second initial class information. For example, the passenger detection device 1000 may include second initial class information (e.g., a probability value that a passenger (e.g., adult, baby) is riding in the second location) related to the second location (e.g., the left position of the rear seat) and 2 It can be compared with the second condition of the rule node, and it can be determined whether the second initial class information satisfies the second condition. For example, the second condition may be a condition that the second initial class information is greater than, equal to, or less than the second threshold probability value, and in this case, the second initial class information (e.g., in the form of a probability value) satisfies the second condition. In this case, the passenger detection device 1000 may determine a classification value related to class information through the second leaf node below the second rule node. On the other hand, if the second initial class information (e.g., in the form of a probability value) does not satisfy the second condition, the passenger detection device 1000 generates a classification value related to the class information through the third leaf node below the second rule node. can be decided.

In the step of determining class information related to the state of the rear seat of the vehicle based on the comparison result (S2330), the passenger detection device 1000 determines the passenger in the rear seat of the vehicle based on the classification value determined through the leaf node based on the comparison result. You can obtain class information indicating whether to board this flight. As an example, the passenger detection device 1000 may determine that the state of the rear seat of the vehicle is the first state (eg, no passenger is on board) based on the classification value classified in the first leaf node. As another example, the passenger detection device 1000 may determine that the state of the rear seat of the vehicle is a second state (eg, the passenger is riding on the left side of the rear seat) based on the classification value classified in the second leaf node. As another example, the passenger detection device 1000 may determine that the state of the rear seat of the vehicle is the third state (eg, the passenger is riding on the right side of the rear seat) based on the classification value classified in the third leaf node.

Meanwhile, the passenger detection device 1000 may determine class information related to the state of the rear seats of the vehicle through analysis of frames for a certain period of time. For example, the passenger detection device 1000 determines that the proportion of frames in which it is determined that a passenger is riding in the rear seat of the vehicle is a certain percentage (e.g., 50%) among the frames for a certain period of time (e.g., a time in the range of 3 to 10 seconds). Only in cases where it is greater than or equal to the number, it can be implemented to ultimately determine that a passenger is riding in the rear seat of the vehicle.

However, the structure of the tree-based classification algorithm and the conditions of each rule node are only examples, and the tree-based classification algorithm has an arbitrary appropriate structure and conditions of rule nodes in order to increase the accuracy of analysis of the status of the rear seats of the vehicle. It can be configured as follows.

A tree-based classification algorithm can optimally learn the shape of nodes (i.e., the structure of the tree-based classification algorithm), the conditions of rule nodes (i.e., critical probability values), and/or the labels of leaf nodes (i.e., classification values). there is. Therefore, according to an embodiment of the present disclosure, it is possible to provide the effect of increasing the accuracy of analysis related to the state of the rear seats of a vehicle using a tree-based classification algorithm.

Referring again to FIG. 3, the passenger detection method according to an embodiment of the present disclosure may include a step (S3000) of analyzing sensor data and detecting biosignals.

In the step S3000 of analyzing sensor data and detecting a biological signal, the passenger detection device 1000 may perform an operation of detecting a biological signal by analyzing sensor data. Specifically, the passenger detection device 1000 may determine whether sensor data satisfies a predetermined rule, and determine whether a biological signal has been detected from the rear seat inside the vehicle based on the determination result. For example, if the sensor data satisfies a predetermined rule, the passenger detection device 1000 may determine that a biosignal has been detected from the rear seat of the vehicle, and if the sensor data does not satisfy the predetermined rule, the passenger detection device 1000 may determine that a biosignal has been detected from the rear seat of the vehicle. It can be determined that biological signals were not detected.

Figure 10 is a flowchart detailing the step (S3000) of detecting a biological signal according to an embodiment of the present disclosure.

The step of detecting a biological signal according to an embodiment of the present disclosure (S3000) includes accumulating the above-described conversion data by a predetermined time window and obtaining the accumulated data (S3100), where the accumulated data is predetermined It may further include a step of determining whether the rule is satisfied (S3200), and a step of determining whether a biological signal is detected from the rear seat inside the vehicle based on the determination result (S3300).

In the step (S3100) of accumulating conversion data by a predetermined time window and acquiring the accumulated data, the passenger detection device 1000 converts sensor data (e.g., conversion data acquired in step S1200) into a predetermined amount. You can accumulate as much as the time window and obtain the accumulated data.

In the step of determining whether the accumulated data satisfies the predetermined rule (S3200), the passenger detection device 1000 may determine whether the accumulated data satisfies the predetermined rule. A predetermined rule may consist of at least one rule. For example, the passenger detection device 1000 may determine whether the accumulated data satisfies a predetermined rule by calculating whether the size of the accumulated data is greater than a predetermined value for the time axis. For example, the passenger detection device 1000 calculates whether the number of times the phase of the accumulated data intersects a specific quantile on the time axis falls within a predetermined range to determine whether the accumulated data satisfies a predetermined rule. You can. For example, the passenger detection device 1000 determines whether the accumulated data satisfies a predetermined rule by calculating whether the difference between the maximum and minimum values on the time axis of the phase value of the accumulated data falls within a predetermined range. can do.

In the step S3300 of determining whether a biological signal has been detected in the rear seat of the vehicle based on the determination result, the passenger detection device 1000 determines whether the accumulated data satisfies predetermined rules through step S32000. Based on this, it can be determined whether biosignals have been detected from the rear seat inside the vehicle. As an example, the passenger detection device 1000 may be implemented to determine that a biological signal has been detected in a range bin corresponding to the accumulated data when there is accumulated data that satisfies the above-mentioned predetermined rule. As another example, when the above-described accumulated data does not satisfy a predetermined rule, the passenger detection device 1000 may be implemented to determine that a biological signal has not been detected in the range bin corresponding to the accumulated data. However, this is only an example, and the passenger detection device 1000 may be implemented to determine whether or not a biometric signal is detected using any appropriate method in order to increase the accuracy of biometric signal analysis.

Referring again to FIG. 3, the passenger detection method according to an embodiment of the present disclosure may include a step (S4000) of detecting movement by analyzing sensor data.

In the step of detecting movement by analyzing sensor data (S4000), the passenger detection device 1000 may perform an operation of detecting movement inside the vehicle by analyzing sensor data. Specifically, the passenger detection device 1000 may determine whether there is movement inside the vehicle based on a value corresponding to the strength of the signal of sensor data. Here, 'movement' means not only the movement of people, but also the movement of arbitrary objects (e.g., animals, people, objects, etc.) that occur inside the vehicle.

Figure 11 is a flow chart specifying the step (S4000) of detecting movement according to an embodiment of the present disclosure.

The step of detecting movement according to an embodiment of the present disclosure (S4000) includes calculating the average value of values corresponding to the intensity of the signal of at least one of the above-described first heat map and the second heat map (S4100), It may further include obtaining a determined threshold value (S4200), and comparing the average value with the threshold value, and determining that movement has been detected inside the vehicle if the average value is greater than or equal to the threshold value (S4300).

In the step (S4100) of calculating the average value of values corresponding to the intensity of the signal of at least one of the first heat map and the second heat map, the passenger detection device 1000 generates a first heat map related to the second direction of the spherical coordinate system. Alternatively, a second heat map related to the third direction of the spherical coordinate system may be obtained, and the average value of values corresponding to the signal intensity of each heat map may be calculated. For example, the passenger detection device 1000 may calculate the average value of values corresponding to the signal strength of the entire area of the first heat map or a partial area of the first heat map. For example, the passenger detection device 1000 may calculate the average value of values corresponding to the signal strength of the entire area of the second heat map or a partial area of the second heat map.

In the step of acquiring a predetermined threshold (S4200), the passenger detection device 1000 may acquire the predetermined threshold. Here, the threshold may be a value corresponding to a value related to the signal strength of the heatmap.

In the step (S4300) of comparing the average value and the threshold value and determining that movement has been detected inside the vehicle if the average value is greater than or equal to the threshold value, the passenger detection device 1000 determines the average value of the values related to the signal strength of the heatmap. and a predetermined threshold, and based on the comparison result, movement inside the vehicle can be detected. For example, the passenger detection device 1000 may determine that movement has been detected inside the vehicle when the average value of the values related to the signal strength of the heat map is greater than or equal to a predetermined threshold. For another example, the passenger detection device 1000 may determine that no movement has been detected inside the vehicle when the average value of the values related to the signal strength of the heat map is less than a predetermined threshold. As an example, the passenger detection device 1000 may be implemented to finally determine movement inside the vehicle based on the result of determining movement based on the first heat map and the result of determining movement based on the second heat map. there is. For example, the passenger detection device 1000 assigns a first weight to the result of determining whether there is movement based on the first heat map, and assigns a second weight to the result of determining whether there is movement based on the second heat map to determine whether the passenger is moving inside the vehicle. It can be implemented to finally determine the movement of .

Referring again to FIG. 3, the passenger detection method according to an embodiment of the present disclosure may include a step (S5000) of determining whether a passenger is on board.

In the step of determining whether a passenger is boarded (S5000), the passenger detection device 1000 may determine whether a passenger is boarded inside the vehicle (eg, in the rear seat). Specifically, the passenger detection device 1000 will be implemented to determine whether or not a passenger is boarding based on class information acquired through a fully learned classification model or determined through a tree-based classification algorithm, whether biometric signals are detected, and whether movement is detected. You can.

According to one embodiment of the present disclosure, the passenger detection device 1000 obtains class information as a second class or a third class (i.e., there is a possibility that a passenger (e.g., an adult or a baby) is riding in the rear seat of the vehicle). , If a biological signal is detected, it can be implemented to determine that a passenger is in the vehicle and make a decision to detect the passenger. For example, in

cases

4 and 6 of FIG. 12 , the passenger detection device 1000 may be implemented to determine that a passenger is in the vehicle and make a decision to detect the passenger.

According to one embodiment of the present disclosure, the passenger detection device 1000 obtains class information as a second class or a third class (i.e., there is a possibility that a passenger (e.g., an adult or a baby) is riding in the rear seat of the vehicle). , If biosignals are not detected, it can be implemented to determine that there are no passengers in the vehicle and make a decision not to detect passengers. For example, the passenger detection device 1000 may be implemented to determine that no passengers are in the vehicle in

cases

3 and 5 of FIG. 12 and to make a decision not to detect passengers.

According to an embodiment of the present disclosure, when movement is detected (e.g., cases 7 to 12 of FIG. 12), the passenger detection device 1000 detects the inside of the vehicle regardless of the class information and the detection result of the biosignal. It can be implemented to determine that the movement of an arbitrary object, including a person, has been detected and to make a motion detection decision.

According to an embodiment of the present disclosure, the passenger detection device 1000 is configured to obtain class information as a first class (i.e., there is a possibility that there is no passenger riding in the rear seat of the vehicle) and no biometric signal is detected (e.g. , Case 1 in FIG. 12) may be implemented to determine that there are no passengers in the vehicle and make a decision not to detect passengers.

According to an embodiment of the present disclosure, the passenger detection device 1000 is configured to detect class information as a first class (i.e., there is a possibility that there is no passenger riding in the rear seat of the vehicle) and a biometric signal is detected (e.g., In case 2 of FIG. 12), it may be implemented to determine that there are no passengers in the vehicle and make a decision not to detect passengers. However, in this case, the passenger detection device 1000 may be implemented to re-perform the analysis or additionally compare the probability value of the class information and the detection probability of the biometric signal to determine whether the passenger is boarded.

However, the content described in relation to FIG. 12 is merely an example, and the passenger detection device 1000 may be implemented to determine whether or not a passenger is boarding according to any suitable method.

Various operations of the passenger detection device 1000 described above may be stored in the memory 1200 of the passenger detection device 1000, and the processor 1300 of the passenger detection device 1000 performs the operations stored in the memory 1200. may be provided to do so.

According to the passenger detection method, passenger detection device, and passenger detection system according to an embodiment of the present disclosure, the accuracy of analysis related to the state of the rear seats of the vehicle can be increased through a tree-based classification algorithm.

Meanwhile, the passenger detection method, passenger detection device, and passenger detection system disclosed in the present disclosure may be installed and utilized in environments where cameras cannot be installed due to security or privacy issues, including public restrooms and nursing homes.

The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present disclosure and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified and implemented in other embodiments by a person with ordinary knowledge in the field to which the embodiments belong. Accordingly, contents related to such combinations and modifications should be construed as being included in the scope of the present disclosure.

In addition, although the above description focuses on the embodiment, this is only an example and does not limit the present disclosure, and those skilled in the art will be able to understand the above without departing from the essential characteristics of the present embodiment. You will see that various modifications and applications not illustrated are possible. In other words, each component specifically shown in the embodiment can be modified and implemented. And these variations and differences related to application should be construed as being included in the scope of the present disclosure as defined in the attached claims.

The passenger detection method, passenger detection device, and passenger detection system described above may be applied to fields related to autonomous vehicles or school vehicles.

Claims

In the method of detecting a passenger by a passenger detection device that analyzes a radar sensor to determine whether a passenger is in the vehicle,

Obtaining sensor data in which electromagnetic waves transmitted through a radar sensor mounted inside the vehicle are reflected by objects inside the vehicle;

Obtaining class information related to the status of the seat of the vehicle from the sensor data through a classification model on which learning has been completed - the class information indicates that the seat of the vehicle is empty Class, a second class meaning that an adult is riding in the seat of the vehicle, and a third class meaning that a baby is riding in the seat of the vehicle;

detecting biological signals by analyzing the sensor data;

detecting movement by analyzing the sensor data; and

A step of determining whether to board a passenger based on the class information, biometric signal detection results, and movement detection results,

The step of obtaining class information related to the status of the seat of the vehicle includes,

Obtaining first initial class information of a first position of a seat of the vehicle through the classification model;

Obtaining second initial class information of a second position of the seat of the vehicle through the classification model; and

Comprising: calculating the class information based on the first initial class information and the second initial class information using a tree-based classification algorithm.

Passenger detection method.
According to claim 1,

The step of determining whether the passenger is boarding is,

When the class information is obtained as the second class or the third class and the biosignal is detected, it is determined that a passenger has boarded the vehicle,

If the class information is obtained as the second class or the third class and the biosignal is not detected, determining that there is no passenger in the vehicle,

When the movement is detected, determining that movement of an object, including a person, has been detected inside the vehicle regardless of the class information and the detection result of the biosignal.

Passenger detection method.
According to claim 1,

The classification model for which the learning has been completed is,

configured to receive the sensor data and output a probability value corresponding to at least one of the first class, the second class, and the third class of the class information based on the sensor data,

Passenger detection method.
According to claim 1,

The classification model for which the learning has been completed is,

It includes an input layer that receives learning data consisting of sensor data and labels assigned to the sensor data, an output layer that outputs a predicted value, and a hidden layer that connects the input layer and the output layer,

Trained by adjusting the weight of at least one node constituting the hidden layer based on the difference between the predicted value and the label,

Passenger detection method.
According to claim 1,

The step of calculating the class information is,

Comparing first initial class information at the first location and a first condition of the first rule node through a first rule node included in the tree-based classification algorithm;

Comparing second initial class information at the second location and a second condition of the second rule node through a second rule node included in the tree-based classification algorithm and connected to the first rule node; and

Determining class information related to the state of the seat of the vehicle based on the comparison result; further comprising,

Passenger detection method.
According to claim 1,

The step of acquiring the sensor data is,

Obtaining raw data reflected by an object inside the vehicle;

performing a Fast Fourier Transform on the raw data in a first direction of a spherical coordinate system to transform the raw data and obtain transformed data; and

acquiring the sensor data by performing capon beamforming on the converted data to obtain a first heat map related to a second direction of a spherical coordinate system and a second heat map related to a third direction of a spherical coordinate system; Including,

Passenger detection method.
According to clause 6,

The step of detecting the biosignal is,

Accumulating the conversion data by a predetermined time window and obtaining accumulated data;

determining whether the accumulated data satisfies a predetermined rule; and

Further comprising: determining whether a biological signal is detected in a seat inside the vehicle based on the determination result,

Passenger detection method.
According to clause 6,

The step of detecting the movement is,

calculating an average value of values corresponding to the intensity of a signal of at least one of the first heat map and the second heat map;

obtaining a predetermined threshold; and

Comparing the average value and the threshold value, and determining that movement has been detected inside the vehicle if the average value is greater than or equal to the threshold value, further comprising:

Passenger detection method.
A computer-readable recording medium recording a program for executing the method according to any one of claims 1 to 8 on a computer.
In a passenger detection device that analyzes a radar sensor to determine whether a passenger is in the vehicle,

a transceiver unit that receives sensor data in which electromagnetic waves transmitted through a radar sensor mounted inside the vehicle are reflected by objects inside the vehicle; and

Including; a processor configured to analyze the sensor data to determine whether a passenger is on board the vehicle;

The processor,

Obtain the sensor data and class information related to the status of the seat of the vehicle from the sensor data through a fully learned classification model - the class information means that the seat of the vehicle is empty associated with a probability value corresponding to at least one of the first class, meaning that an adult is riding in the seat of the vehicle, and the third class, meaning that an adult is riding in the seat of the vehicle. acquire, analyze the sensor data to detect bio-signals, analyze the sensor data to detect movement, and determine whether or not a passenger is boarding based on the class information, bio-signal detection results, and movement detection results. It is composed,

The processor,

Obtaining first initial class information of the first position of the seat of the vehicle through the classification model, obtaining second initial class information of the second position of the seat of the vehicle through the classification model, and tree-based classification algorithm Configured to obtain class information related to the state of the seat of the vehicle by calculating the class information based on the first initial class information and the second initial class information using

Passenger detection device.