WO2021251637A1 - Signal processing apparatus and method using vehicle radar sensor, and computer-readable recording medium - Google Patents

Abstract

Provided are a signal processing apparatus and method using a vehicle radar sensor, and a computer-readable recording medium. The signal processing apparatus using the vehicle radar sensor may comprise: an RF module which generates a beat frequency signal that is a difference frequency between a transmitted FMCW signal and an FMCW signal received by being reflected by an object whose movement or position including a movement changes over time; a signal processing module which generates an image having a micro-Doppler frequency change pattern on the basis of the generated beat frequency signal; and a CNN module which classifies the object according to the generated image having the micro-Doppler frequency change pattern.

Description

Signal processing apparatus, method and computer-readable recording medium using vehicle radar sensor

The present application relates to a signal processing apparatus using a vehicle radar sensor, a method, and a computer-readable recording medium.

Object detection/classification/recognition using vehicle sensors is important information of the Advanced Driver Assistant System (ADAS), and active responses can be made according to the classification/recognition of objects (eg, pedestrians, motorcycles, trucks) .

Current object classification uses image sensors (eg digital cameras) to classify objects. However, in the image sensor, object classification or recognition performance is significantly degraded depending on weather conditions (eg, snow, rain, fog) and day/night (eg, sunrise, sunset, night).

On the other hand, since the radar sensor uses electromagnetic waves, the performance degradation due to bad weather, fog, and day/night conditions is insignificant, unlike the image sensor.

Such a radar sensor transmits several frequency-modulated signals, and then analyzes the received signal reflected from the object to detect the object's position (e.g., distance and angle between the radar sensor and the object) and speed (e.g., Doppler frequency). .

The Doppler frequency refers to a frequency generated by the speed of an object when a signal transmitted from a radar sensor is reflected by a moving object.

The currently developed object classification method using a vehicle radar sensor is a method of improving the distance resolution using a high bandwidth, which is referred to as a 'High Resolution Range Profile (HRRP)' (see FIG. 1 ).

However, this method has a problem in that it requires a large number of discrete samples at a high sampling frequency, and a large amount of calculation is required.

As a related technology, there is Korean Patent Application Laid-Open No. 2018-0042052 ('A device and method for detecting a target using an FMCW radar, publication date: April 25, 2018).

The present invention provides a signal processing apparatus using a vehicle radar sensor capable of classifying an object with a small amount of calculation, a method, and a computer-readable recording medium.

According to an embodiment of the present invention, a beat frequency signal that is a difference frequency between a transmitted FMCW (Frequency Modulation Continuous Wave) signal and a received FMCW signal reflected by an object whose position including movement or movement is changed according to time RF module to generate; a signal processing module for generating an image of a micro-Doppler frequency change pattern based on the generated beat frequency signal; Convolutional Neural Network (CNN) module for classifying the object according to the generated image of the micro-Doppler frequency change pattern; provides a signal processing apparatus using a vehicle radar, including a.

According to an embodiment of the present invention, in the RF module, the transmitted FMCW (Frequency Modulation Continuous Wave) signal and the difference frequency of the received FMCW signal reflected by an object whose position including movement or movement is changed according to time a first step of generating an in-bit frequency signal; a second step of generating, in a signal processing module, an image of a micro-Doppler frequency change pattern based on the generated beat frequency signal; and a third step of classifying the object according to the generated image of the micro-Doppler frequency change pattern in a Convolution Neural Network (CNN) module.

According to one embodiment of the present invention, there is provided a computer-readable recording medium in which a program for executing the method in a computer is recorded.

According to an embodiment of the present invention, a micro Doppler frequency change pattern generated based on a difference frequency between a transmitted FMCW signal and a received FMCW signal reflected by an object whose position including movement or movement changes with time By classifying the object according to the image of

1 is a diagram for explaining object classification using a conventional high-resolution distance profile.

2 is a block diagram of a signal processing apparatus using a vehicle radar sensor according to an embodiment of the present invention.

3 is a diagram illustrating a transmitted FMCW signal and a received FMCW signal reflected by an object according to an embodiment of the present invention.

4 is a diagram for explaining a process of extracting a range bin in which an object exists among a plurality of range bins in which a difference frequency according to a delay time is converted into a distance according to an embodiment of the present invention.

FIG. 5 is a view for explaining a process of obtaining a modified angle when a plurality of reception channels are included according to an embodiment of the present invention.

6 is a diagram illustrating a location of an object displayed on a distance-angle map through location tracking of the object according to an embodiment of the present invention.

7 is a micro-Doppler frequency change pattern generated from a plurality of chirps included in a plurality of frames of an FMCW signal collected with respect to an object tracked using an extended Kalman filter according to an embodiment of the present invention and beat frequency signals thereof; It is a drawing showing an image of

8 is an image of a micro Doppler frequency change pattern generated using a plurality of frames of an FMCW signal according to an embodiment of the present invention and an image of a micro Doppler frequency change pattern generated using one frame of an FMCW signal. It is the drawing shown.

9 is a diagram illustrating layers constituting a convolutional neural network model according to an embodiment of the present invention.

10 is a flowchart illustrating a signal processing method using a vehicle radar sensor according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the embodiment of the present invention may be modified in various other forms, and the scope of the present invention is not limited only to the embodiments described below. The shapes and sizes of elements in the drawings may be exaggerated for a clearer description, and elements indicated by the same reference numerals in the drawings are the same elements.

2 is a block diagram of a signal processing apparatus 100 using a vehicle radar sensor according to an embodiment of the present invention.

First, as shown in FIG. 2 , in the signal processing apparatus 100 using the vehicle radar sensor according to an embodiment of the present invention, any one of a position including movement or movement according to a transmitted FMCW signal and time is changed. An RF module 110 that generates a beat frequency signal that is a difference frequency of the received FMCW signal reflected by the object S, and a signal processing module that generates an image of a micro Doppler frequency change pattern based on the generated beat frequency signal 123 and a Convolution Neural Network (CNN) module 124 that classifies the object S according to the image of the generated micro-Doppler frequency change pattern, and a signal processing module 123 and a CNN module 124 ) between the low-pass filter 121 for low-pass filtering the bit frequency signal and the A/D converter 122 for converting the low-pass filtered bit frequency signal into a digital signal may be further provided. For example, when the object S is a pedestrian, the movement may refer to movement of an arm or leg, and the position may refer to a position according to the movement of the pedestrian.

First, the RF module 110 transmits the FMCW signal in response to the frequency modulation control signal output from the signal processing module 123, and the bit frequency signal that is the difference frequency of the received FMCW signal reflected by the object S. It may be generated and transmitted to the signal processing module 123 . The RF module 110 may include a waveform generator 111 , a voltage controlled oscillator 112 , a power amplifier 113 , a low noise amplifier 114 , and a frequency synthesizer 115 .

Specifically, the waveform generator 111 may generate a signal corresponding to the frequency modulation control signal output from the signal processing module 123 and having information about the frequency modulation of the transmission signal.

The voltage-controlled oscillator 112 may generate an oscillation signal having a constant frequency in response to a signal output from the waveform generator 111 .

The power amplifier 113 may amplify the oscillation signal generated from the voltage controlled oscillator 112 .

The low-noise amplifier 114 may amplify only a true received signal excluding noise included in the received FMCW signal.

The frequency synthesizer 115 may generate a synthesized signal by adding the received signal amplified by the low noise amplifier 114 and the oscillating signal output from the voltage controlled oscillator 112 .

Here, the reason for synthesizing the received signal and the oscillation signal is to generate a bit frequency signal expressed as a frequency difference between the transmit signal and the received signal.

The low-pass filter 121 may remove noise of a high-frequency component by filtering only a signal of a low frequency band among the bit frequency signals output from the RF module 110 .

The A/D converter 122 may convert the low-pass filtered bit frequency signal into a digital signal corresponding thereto.

A transmission signal according to an embodiment of the present invention uses an FMCW signal. For this, one frequency-modulated signal is defined as a chirp, and a plurality of chirp signals are transmitted at regular time intervals for a predetermined time. Also, an arrangement of a plurality of chirped signals for a predetermined time is defined as one frame.

On the other hand, the signal processing module 123 outputs a frequency modulation control signal necessary for generating a transmission waveform to the waveform generator 111, and based on the beat frequency signal generated by the frequency synthesizer 115, an image of a micro Doppler frequency change pattern. can create

To this end, the signal processing module 123 calculates a plurality of range bins in which a difference frequency according to a delay time is converted into a distance for each of a plurality of chirp signals received in one frame for a single time. In addition, a range bin in which an object (S) exists among a plurality of range bins for each chirp signal is extracted, and a microprocessor through Short Time Fourier Transform (STFT) of a beat frequency signal corresponding to the extracted range bin An image of a Doppler frequency change pattern can be generated.

Here, when an image of a micro-Doppler frequency change pattern is generated using a plurality of beat frequency signals sensed in one frame, it is difficult to classify an object because it has a low-resolution image. Therefore, according to an embodiment of the present invention, a plurality of chirped signals included in a plurality of frames are received, and the corresponding beat frequency is obtained by tracking the object according to the changed distance and angle of the object S according to the continuous time change. can be collected Thereafter, an image of a high-resolution micro-Doppler frequency change pattern may be generated through a Short Time Fourier Transform (STFT) of a plurality of beat frequencies collected using a plurality of frames. This will be described later with reference to FIG. 7 .

Specifically, the signal processing module 123 frequency-transforms (Fast Fourier Transform, FFT) one bit frequency signal in one frame received from the frequency synthesizer 115, and the size of the frequency-converted bit frequency signal for each distance. to calculate This is repeated for a plurality of beat frequency signals. Thereafter, when the frequency-converted bit frequency signal has a size greater than or equal to a predetermined threshold, a range bin to which the generated beat frequency signal belongs may be extracted as a range bin in which an object exists.

3 is a diagram illustrating a transmitted FMCW signal and a received FMCW signal reflected by an object according to an embodiment of the present invention, and FIG. 4 is a distance difference frequency according to a delay time according to an embodiment of the present invention A diagram for explaining a process of extracting a range bin in which an object exists among a plurality of range bins converted to .

Hereinafter, a process of extracting a range bin having an object from among a plurality of range bins will be described in detail with reference to FIGS. 3 and 4 .

3, in the present invention, the FMCW signal uses a sawtooth wave shape whose frequency increases with time, and the transmit FMCW signal (Tx Signal) shown in the solid line and the received FMCW signal (Rx Signal) shown in the dotted line are used. There is a difference frequency (fbeat, or 'beat frequency') equal to the delay time τ. Here, fsweep is the bandwidth, Tchip is the time length of the chirp, and 1 to L _N are the number of chirps transmitted per frame. Although FIG. 3 illustrates a sawtooth wave, the present invention is not necessarily limited thereto, and it goes without saying that it can also be applied to a triangular wave.

3, the beat frequency fbeat is proportional to the distance R to the object S, and can be expressed by Equation 1 below.

where fbeat is the beat frequency, c is the speed of light, fsweep is the bandwidth, Tchirp is the length of time of the chirp, and R is the distance to the object.

Meanwhile, as shown in FIG. 4 , the signal processing module 123 may extract a range bin in which the object S exists among a plurality of range bins obtained by converting a difference frequency according to a delay time into a distance.

Specifically, (a) of FIG. 4 shows a plurality of range bins (range bin #1 to range bin #256) in which a difference frequency fbeat according to a delay time (τ in FIG. 3 ) according to Equation 1 is converted into a distance. 4 (b) shows the frequency conversion of the bit frequency signal, which is the difference frequency between the transmitted FMCW signal and the received FMCW signal reflected by the object, where the X axis is the bit frequency and the Y axis is the signal size.

That is, the beat frequency signal generated by the frequency mixer 115 is in the form of a square wave, and when it is frequency-converted and expressed in the frequency domain, as shown in FIG. can

Accordingly, the signal processing module 123 frequency-transforms (FFT) the bit frequency signal received from the frequency synthesizer 115 , and when the magnitude of the frequency-converted bit frequency signal is greater than or equal to a predetermined threshold value, the generated bit frequency signal belongs to A range bin (range bin #2 in FIG. 4 ) may be extracted as a range bin in which an object exists. Here, the threshold value may be a preset value.

On the other hand, according to an embodiment of the present invention, the signal processing module 123 frequency-converts the bit frequency signals of each of the plurality of chirps included in one frame of the FMCW signal, When the sum value (refer to FIG. 4(c) ) is equal to or greater than a predetermined threshold, a range bin to which the generated beat frequency signal belongs may be extracted as a range bin in which an object exists.

As such, the reason for summing the magnitudes of the plurality of frequency-converted beat frequency signals is that the magnitude of one frequency-converted bit frequency signal may not have a sufficient peak value to detect the object S.

On the other hand, according to an embodiment of the present invention, when the RF module 110 includes a plurality of reception channels, the signal processing module 123 obtains the reception angle of the FMCW signal with respect to the range bin in which the object is detected, A range bin in which a reception angle and an object are present may be plotted on a distance-angle map indicating a reception angle compared to the range bin.

FIG. 5 is a diagram illustrating a process for obtaining a mathematical expression angle and a distance-angle map when a plurality of reception channels are included according to an embodiment of the present invention. In (a) of FIG. 5, the chirps received through a plurality of receiving antennas (here, four) are Rx#0 to Rx#3, respectively, the distance between the receiving antennas is d, and the chirps received through the four receiving antennas are d. The frequency-converted values of the bit frequency signals for each are expressed as Z ₀ [2] to Z ₃ [2]. Meanwhile, in FIG. 5B , when an object is detected in range bin #2, it is displayed on the distance-angle map.

That is, as shown in (a) of Figure 5, the frequency-converted value of the bit frequency signal for each chirp received through the four reception antennas is Z ₀ [2] to Z ₃ [2], The angle can be obtained according to Equation 2 below.

where H ^H () is the system matrix, Δθ is the angular resolution, i is from -60 to 60 degrees in units of 1 degree, λ is the center frequency wavelength, d is the distance between the receiving antennas, Z ₀ [2] to Z ₃ [2] is a frequency-converted value of a bit frequency signal for each chirp received through four receiving antennas.

That is, as shown in FIG. 5 , when the RF module 110 includes a plurality of reception channels, the signal processing module 123 obtains a reception angle of the FMCW signal with respect to a range bin in which an object is detected, and the obtained A range bin in which a reception angle and an object are present may be plotted on a distance-angle map indicating a reception angle compared to the range bin.

In particular, according to an embodiment of the present invention, the signal processing module 123 obtains a reception angle only for the first chirp (Chirp #1) among a plurality of chirps included in one frame, and one frame receives 90 chirps. If included, since it is usually less than 1 ms to transmit and receive 90 chirps, it can be seen that there is little change in the angle of the object during 1 ms. .

Meanwhile, according to an embodiment of the present invention, the signal processing module 123 tracks the position of the object using the extended Kalman filter based on the distance-angle map per frame, and sets the position of the tracked object in each frame. It can be plotted on a per distance-angle map.

6 is a diagram illustrating a location of an object displayed on a distance-angle map through location tracking of the object according to an embodiment of the present invention.

As shown in FIG. 6, when a pedestrian (object) moves, the FMCW signal of one frame (consisting of 90 chirps) is transmitted and received at k-2 time point, k-1 time point, and time point k, respectively, and the signal processing module ( 123) may track the location of the object using the extended Kalman filter, and display the tracked location of the object on the distance-angle map. The extended Kalman filter for tracking the position of an object is a well-known technique, and thus, it will not be described in detail in the present invention. In addition, by using the extended Kalman filter to track the position of the object over time, it is possible to collect the corresponding bit frequency of each of a plurality of chirps included in each frame over a plurality of frames of the FMCW signal.

Thereafter, the signal processing module 123 performs a high-resolution micro Doppler frequency change pattern image through a Short Time Fourier Transform (STFT) of a bit frequency signal corresponding to a range bin in which the position of the object is tracked according to time. can create In the present invention, an image of a high-resolution micro Doppler frequency change pattern may mean an image of a micro Doppler frequency change pattern generated with at least 512 or more bit frequency samples.

In particular, according to an embodiment of the present invention, the signal processing module 123 is a bit frequency signal of each of a plurality of chirps included in each frame over a plurality of frames of an FMCW signal for an object tracked using the extended Kalman filter. An image of a micro-Doppler frequency change pattern can be generated through a short-time Fourier transform (STFT) of

In relation to this, FIG. 7 shows a plurality of chirps included in a plurality of frames of an FMCW signal collected with respect to an object tracked using an extended Kalman filter according to an embodiment of the present invention and a micro Doppler generated from the beat frequency signals thereof. It is a diagram showing an image of a frequency change pattern.

That is, as shown in FIG. 7 , the signal processing module 123 uses the extended Kalman filter to track the position of the object according to the time passage, and to each frame over a plurality of frames (frame #1 to frame #k). A beat frequency signal of each of the included chirps (chirp #1 to chirp #N) is collected. Thereafter, an image of a micro-Doppler frequency change pattern as shown in FIG. 7B may be generated through a short-time Fourier transform (STFT) of each of the plurality of chirps of the plurality of collected frames.

In FIG. 7 , the distance of an object tracked using the extended Kalman filter during k, which is the number of frames, is the same. do. Also, in the present invention, the number of chirps, N, may be, for example, 90, and the number of frames, k, may be 10, but it should be noted that it is not necessarily limited to a specific numerical value.

Meanwhile, FIG. 8 is an image of a micro Doppler frequency variation pattern generated using a plurality of frames of an FMCW signal and an image of a micro Doppler frequency variation pattern generated using one frame of an FMCW signal according to an embodiment of the present invention. is a diagram showing comparison. Here, (a) is a color image of a micro Doppler frequency variation pattern generated through a plurality of frames, and (b) is a color image of a micro Doppler frequency variation pattern generated through one frame.

As shown in FIG. 8 , the higher the number of samples for the short-time Fourier transform (STFT), the higher the frequency resolution. It can be seen that the image of the micro-Doppler frequency change pattern generated through the frame of .

Referring back to Figure 2, the image of the micro-Doppler frequency change pattern generated by the signal processing module 123 is input to the CNN module 124, and the CNN module 124 is the image of the input micro-Doppler frequency change pattern. Depending on the object (S) can be classified.

The CNN (Convolution Neural Network) module 124 includes a Doppler feature extraction module 124a that extracts micro Doppler pattern features by applying a pre-learned convolutional neural network model to an image of a micro Doppler frequency change pattern, and a Doppler feature extraction module ( A classification module 124b for classifying an object as a pedestrian or a non-pedestrian according to the micro-Doppler pattern feature extracted in 124a) may be included.

9 is a diagram illustrating layers constituting a convolutional neural network model according to an embodiment of the present invention.

The convolutional neural network model is a type of deep neural network (DNN), and includes seven convolutional layer groups (G1 to G7) including at least one convolutional layer, an average pool layer , AVG pool), a fully connected layer (FC: 2), and a softmax layer. It has a structure suitable for learning two-dimensional data, and is can be trained The convolutional neural network model according to an embodiment of the present invention may be referred to as 'Resnet 14'.

According to an embodiment of the present invention, a group of 7 convolutional layers moves 16 filters of a size of 5×5×3 in an image of a size of 100×91×3 pixels in the vertical, left, and right directions at intervals of 1 pixel by 2 padding. A first convolutional layer group 811 and G1 for extracting a micro-Doppler pattern feature map having a size of 100×91×16 may be included.

According to an embodiment of the present invention, the seven convolutional layer groups include a filter 16 having a size of 3×3×16 in a micro Doppler pattern feature map having a size of 100×91×16 output through the first convolutional layer group G1. A first convolutional layer 812 that extracts a micro-Doppler pattern feature map with a size of 100×91×16 by moving the dog one padding at intervals of 1 pixel in the up, down, left and right directions, and the first convolutional layer 812 In the 100×91×16 size micro Doppler pattern feature map, 16 3×3×16 filters are moved up, down, left and right by one padding at 1 pixel intervals to create a 100×91×16 micro Doppler pattern feature map. The second convolutional layer group G2 including the extracted second convolutional layer 813 may be included.

According to an embodiment of the present invention, the seven convolutional layer groups include a filter 16 having a size of 3×3×16 in a micro Doppler pattern feature map having a size of 100×91×16 output through the second convolutional layer group G2. A first convolutional layer 814 that extracts a micro Doppler pattern feature map with a size of 100×91×16 by moving the dog one padding at intervals of 1 pixel in the up, down, left and right directions, and the first convolutional layer 814 Output through In the 100×91×16 sized micro Doppler pattern feature map, 16 3×3×16 filters are moved up, down, left and right by one padding at 1 pixel intervals to create a 100×91×16 micro Doppler pattern feature map. may include a third convolutional layer group G3 including a second convolutional layer 815 for extracting .

According to an embodiment of the present invention, the seven convolutional layer groups include a filter 32 having a size of 3×3×16 in a micro Doppler pattern feature map having a size of 100×91×16 output through the third convolution layer group G3. The first convolutional layer 816 that extracts the micro Doppler pattern feature map of 50×46×32 size by moving the dog one padding at an interval of 2 pixels in the up, down, left and right directions, and the first convolutional layer 816 In the 50×46×32 micro Doppler pattern feature map, 32 filters of 3×3×32 size are moved up, down, left and right by one padding at 1 pixel intervals to create a 50×46×32 micro Doppler pattern feature map. A fourth convolutional layer group G4 including the extracted second convolutional layer 817 may be included.

According to an embodiment of the present invention, the seven convolutional layer groups include a 3×3×32 filter 32 in the 50×46×32 micro Doppler pattern feature map output through the fourth convolutional layer group G4. A first convolutional layer 818 that extracts a 50×46×32 micro Doppler pattern feature map by moving the dog one padding at 1 pixel intervals in the vertical, left and right directions, and the first convolutional layer 818 In the 50×46×32 micro Doppler pattern feature map, 32 filters of 3×3×32 size are moved up, down, left and right by one padding at 1 pixel intervals to create a 50×46×32 micro Doppler pattern feature map. A fifth convolutional layer group G5 including the second convolutional layer 819 to be extracted may be included.

According to an embodiment of the present invention, the seven convolutional layer groups include a filter 64 having a size of 3×3×32 in a micro Doppler pattern feature map having a size of 50×46×32 output through the fifth convolutional layer group G5. A first convolutional layer 820 that extracts a micro-Doppler pattern feature map having a size of 25×23×64 by moving the dog one padding at an interval of 2 pixels in the up, down, left, and right directions, and the 25 output through the first convolutional layer 820 In the ×23×64 size micro Doppler pattern feature map, 64 3×3×32 filters are moved up, down, left, and right by one padding at 1 pixel intervals to extract a 25×23×64 micro Doppler pattern feature map. A sixth convolutional layer group G6 including a second convolutional layer 821 may be included.

According to an embodiment of the present invention, the seven convolutional layer groups include a filter 64 having a size of 3×3×64 in a micro Doppler pattern feature map having a size of 25×23×64 output through the sixth convolutional layer group G6. A first convolutional layer 822 that extracts a micro Doppler pattern feature map having a size of 25×23×64 by moving the dog one padding at intervals of 1 pixel in the up, down, left and right directions, and the first convolutional layer 822 In the 25×23×64 micro Doppler pattern feature map, 64 filters of 3×3×32 size are moved up, down, left, and right by one padding at 1 pixel intervals to create a 25×23×64 micro Doppler pattern feature map. A seventh convolutional layer group G7 including the extracted second convolutional layer 823 may be included.

In addition, according to an embodiment of the present invention, the second convolutional layer group G2 to the seventh convolutional layer group G6 include the micro Doppler pattern feature map extracted in the previous step and the micro Doppler pattern feature extracted in the current step. Information for each element of the map can be combined through Short Cut (SC).

Meanwhile, according to an embodiment of the present invention, the average full layer 824 has a size of 7×7×64 in the micro Doppler pattern feature map of size 25×23×64 output through the seventh convolutional layer group G7. A 1×1×64 size micro Doppler pattern feature map can be extracted by moving one filter in the vertical direction by one padding at 1-pixel intervals.

And, the fully connected layer 825 is connected to the average full layer 824 and has 64 and 2 nodes as inputs and outputs, respectively, and reduces the micro-micro Doppler pattern feature map using weights of 64×2, , softmax layer may calculate the probability of the micro-micro Doppler pattern feature map output through the fully connected layer 824 according to Equation 3 below.

here,

is the probability for the i-th class, and has a value between 0 and 1. Z ⁽ⁱ⁾ is the feature value (weight) for the i-th class, and J means the total number of classes. In the present invention, when J=2 and i=1, it can be classified as a pedestrian micro Doppler pattern, and when i=2, it can be classified as a non-pedestrian micro Doppler pattern.

Finally, the classification module 124b may classify the object as a pedestrian or a non-pedestrian according to the calculated probability. That is, if the probability calculated according to Equation 4 is greater than 0.5, it may be determined as a pedestrian, and if it is less than 0.5, it may be determined as not a pedestrian.

Meanwhile, in Table 1 below, the parameters of the above-described Resnet 14 model according to an embodiment of the present invention are summarized.

Meanwhile, cross validation was performed for more accurate analysis of the Resnet-14 model according to an embodiment of the present invention. The cross-validation method used 10-fold Cross Validation, and after randomly dividing 30,252 images of the collected micro-Doppler frequency change patterns into 10 groups, one group per fold was selected for testing. and the remaining 9 groups performed training to calculate the accuracy and loss of cross-validation.

In addition, the Resnet-14 model according to an embodiment of the present invention has a faster learning time because it consists of a smaller number of parameters than the existing Resnet-18 model and the AlexNet model.

Table 2 below shows a comparison of the model parameters of Resnet-14 and the existing Resnet-18 and AlexNet according to an embodiment of the present invention.

As shown in Table 2, it can be seen that the cross-validation accuracy of the Resnet-14 model according to an embodiment of the present invention is similar to that of other models, but the cross-validation loss, one-time processing time, and the number of model parameters are significantly lower. can

As described above, according to an embodiment of the present invention, by classifying an object according to an image of a micro-Doppler frequency change pattern generated based on the difference frequency between the transmitted FMCW signal and the received FMCW signal reflected by the object, There is an advantage in that an object can be classified with a small amount of calculation.

Meanwhile, FIG. 10 is a flowchart illustrating a signal processing method using a vehicle radar sensor according to an embodiment of the present invention.

Hereinafter, a signal processing method using a vehicle radar sensor according to an embodiment of the present invention will be described with reference to FIGS. 1 to 10 . However, descriptions of overlapping parts with respect to FIGS. 1 to 10 will be omitted for the sake of simplification of the invention.

1 to 10 , in the signal processing method using a vehicle radar sensor according to an embodiment of the present invention, the transmitted FMCW signal and the received FMCW signal are reflected by the object S in the RF module 110 . It may be initiated by the step of generating a bit frequency signal that is the difference frequency of the signal (S1000).

Next, the signal processing module 123 may generate an image of the micro-Doppler frequency change pattern based on the generated beat frequency signal (S1010).

In step S1010, according to an embodiment of the present invention, the signal processing module 123 extracts a range bin in which the object S exists among a plurality of range bins obtained by converting a difference frequency according to a delay time into a distance, and extracts As described above, an image of a micro-Doppler frequency change pattern can be generated through a Short Time Fourier Transform (STFT) of a beat frequency signal corresponding to the range bin.

In addition, in step S1010, the signal processing module 123 frequency-transforms the bit frequency signal received from the frequency synthesizer 115 (Fast Fourier Transform, FFT), and the magnitude of the frequency-converted bit frequency signal is set to a predetermined threshold value. As described above, in the above case, the range bin to which the generated beat frequency signal belongs can be extracted as the range bin in which the object exists.

In particular, in step S1010, the signal processing module 123 frequency-converts the bit frequency signals of each of the plurality of chirps included in one frame of the FMCW signal, and the sum of the magnitudes of the frequency-converted plurality of bit frequency signals (FIG. 4). As described above, if (c) of (refer to (c)) is greater than or equal to a predetermined threshold, a range bin to which the generated beat frequency signal belongs can be extracted as a range bin in which an object exists.

In addition, in step S1010, the signal processing module 123 tracks the position of the object according to time by using the extended Kalman filter, and over a plurality of frames of the FMCW signal, each corresponding bit of a plurality of chirps included in each frame As described above, a high-resolution micro-Doppler frequency change pattern image can be generated by collecting a frequency and performing short-time Fourier transform (STFT) of the collected beat frequency signal.

Finally, the CNN module 124 may classify the object S according to the image of the generated micro-Doppler frequency change pattern (S1020).

Specifically, the Doppler feature extraction module 124a of the CNN module 124 extracts the micro Doppler pattern feature by applying a pre-learned convolutional neural network model to the image of the micro Doppler frequency change pattern, and in the CNN module 124 As described above, the classification module 124b may classify the object as a pedestrian or a non-pedestrian according to the extracted micro-Doppler pattern feature.

As described above, according to an embodiment of the present invention, by classifying an object according to an image of a micro-Doppler frequency change pattern generated based on the difference frequency between the transmitted FMCW signal and the received FMCW signal reflected by the object, There is an advantage in that an object can be classified with a small amount of calculation.

The signal processing method using the vehicle radar sensor according to the embodiment of the present invention described above may be produced as a program to be executed by a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium is distributed in a computer system connected to a network, so that the computer-readable code can be stored and executed in a distributed manner. And a functional program, code, and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention pertains.

In addition, in describing the present invention, '~ module' refers to various methods, for example, a processor, program instructions executed by the processor, software module, microcode, computer program product, logic circuit, application-specific integrated circuit, firmware and the like can be implemented.

The present invention is not limited by the above-described embodiments and the accompanying drawings. It is intended to limit the scope of rights by the appended claims, and it is to those of ordinary skill in the art that various types of substitutions, modifications and changes can be made without departing from the technical spirit of the present invention described in the claims. it will be self-evident

Claims (22)

1. An RF module for generating a beat frequency signal that is a difference frequency between the transmitted FMCW (Frequency Modulation Continuous Wave) signal and the received FMCW signal by being reflected by an object whose position including movement or movement according to time is changed;

   a signal processing module for generating an image of a micro-Doppler frequency change pattern based on the generated beat frequency signal; and

   a Convolution Neural Network (CNN) module for classifying the object according to the generated image of the micro-Doppler frequency change pattern;

   Including, a signal processing device using a vehicle radar.
2. According to claim 1,

   The signal processing module,

   extracting a range bin in which the object exists among a plurality of range bins in which a difference frequency according to a delay time is converted into a distance,

   A signal processing apparatus using a vehicle radar to generate an image of the micro-Doppler frequency change pattern through Short Time Fourier Transform (STFT) of a beat frequency signal corresponding to the extracted range bin.
3. 3. The method of claim 2,

   The signal processing module,

   frequency-converting the generated beat frequency signal;

   A signal processing apparatus using a vehicle radar for extracting a range bin to which the generated beat frequency signal belongs as a range bin in which the target exists when the frequency-converted magnitude of the beat frequency signal is greater than or equal to a predetermined threshold.
4. 4. The method of claim 3,

   The signal processing module,

   Frequency-converting the bit frequency signal of each of a plurality of chirps included in one frame of the FMCW signal,

   When the sum of the magnitudes of the frequency-converted plurality of beat frequency signals is equal to or greater than a predetermined threshold, extracting a range bin to which the generated beat frequency signal belongs as a range bin in which the object exists,

   Signal processing device using vehicle radar.
5. 3. The method of claim 2,

   The signal processing module,

   If a plurality of reception channels are included, the reception angle of the FMCW signal is obtained with respect to the range bin in which the object is detected,

   A signal processing apparatus using a vehicle radar for displaying the obtained reception angle and a range bin in which the target exists on a distance-angle map indicating a reception angle compared to the range bin.
6. 6. The method of claim 5,

   The signal processing module,

   A signal processing apparatus using a vehicle radar to obtain the reception angle only for a first chirp among a plurality of chirps included in one frame.
7. 6. The method of claim 5,

   The signal processing module,

   A signal processing apparatus using a vehicle radar for tracking the position of the object using an extended Kalman filter, and displaying the tracked position of the object on the distance-angle map.
8. 3. The method of claim 2,

   The signal processing module,

   The position of the object is tracked using an extended Kalman filter, and the micro Doppler is performed through a short-time Fourier transform of each beat frequency signal of a plurality of chirps included in each frame over a plurality of frames of an FMCW signal based on the tracked position. A signal processing device using a vehicle radar to generate an image of a frequency change pattern.
9. According to claim 1,

   The CNN (Convolution Neural Network) module,

   a Doppler feature extraction module for extracting micro Doppler pattern features by applying a pre-trained convolutional neural network (CNN) model to the generated image of the micro Doppler frequency change pattern; and

   a classification module for classifying the object as a pedestrian or a non-pedestrian according to the micro-Doppler pattern features extracted by the Doppler feature extraction module;

   Including, a signal processing device using a vehicle radar.
10. 10. The method of claim 9,

    The convolutional neural network (CNN) model is

    7 convolutional layer groups containing at least one convolutional layer, including an average full layer, a fully connected layer and a softmax layer,

    At least one convolutional layer group, including a shortcut, a signal processing apparatus using a vehicle radar.
11. 11. The method of claim 10,

    The seven convolutional layer groups are

    A method of extracting a 100×91×16 micro Doppler pattern feature map by moving 16 5×5×3 filters in the up, down, left, and right directions at intervals of 1 pixel from an image of 100×91×3 pixels. 1 A signal processing apparatus using a vehicle radar, including a convolutional layer group.
12. 12. The method of claim 11,

    The seven convolutional layer groups are

    In the micro-Doppler pattern feature map of size 100×91×16 output through the first convolutional layer group, 16 filters of size 3×3×16 are moved up, down, left, and right by one padding at 1-pixel intervals in the 100×91 A first convolutional layer for extracting a micro-Doppler pattern feature map with a size of ×16, and 16 filters of a size of 3×3×16 from a micro-Doppler pattern feature map with a size of 100×91×16 output through the first convolution layer A signal using a vehicle radar, including a second convolutional layer group including a second convolutional layer for extracting a micro Doppler pattern feature map of 100 × 91 × 16 by moving one padding at intervals of 1 pixel in the vertical, left and right directions. processing unit.
13. 13. The method of claim 12,

    The seven convolutional layer groups are

    In the micro-Doppler pattern feature map of size 100×91×16 output through the second convolutional layer group, 16 filters of size 3×3×16 are moved up, down, left, and right by one padding at 1-pixel intervals in the 100×91 A first convolutional layer for extracting a micro-Doppler pattern feature map with a size of ×16, and 16 filters of a size of 3×3×16 from a micro-Doppler pattern feature map with a size of 100×91×16 output through the first convolution layer A signal using a vehicle radar, including a third convolutional layer group including a second convolutional layer for extracting a 100×91×16 micro Doppler pattern feature map by moving 1 padding at 1 pixel intervals in the vertical, left and right directions. processing unit.
14. 14. The method of claim 13,

    The seven convolutional layer groups are

    In the 100×91×16 micro-Doppler pattern feature map output through the third convolutional layer group, 32 filters of 3×3×16 size are moved up, down, left, and right by one padding at 2 pixel intervals in the vertical, left, and right direction to 50×46 A first convolutional layer for extracting a micro-Doppler pattern feature map with a size of ×32, and 32 filters of a size of 3×3×32 from a micro-Doppler pattern feature map with a size of 50×46×32 output through the first convolution layer A signal using a vehicle radar, including a fourth convolutional layer group including a second convolutional layer that extracts a 50×46×32 micro Doppler pattern feature map by moving 1 padding at 1 pixel intervals in the vertical, left and right directions. processing unit.
15. 15. The method of claim 14,

    The seven convolutional layer groups are

    In the 50×46×32 micro-Doppler pattern feature map output through the fourth convolutional layer group, 32 filters of 3×3×32 size are moved up, down, left, and right by one padding at 1-pixel intervals in the 50×46 A first convolutional layer for extracting a micro-Doppler pattern feature map with a size of ×32, and 32 filters of a size of 3×3×32 from a micro-Doppler pattern feature map with a size of 50×46×32 output through the first convolution layer A signal using a vehicle radar, including a fifth convolutional layer group including a second convolutional layer for extracting a 50×46×32 micro Doppler pattern feature map by moving 1 padding at 1 pixel intervals in the vertical and horizontal directions. processing unit.
16. 16. The method of claim 15,

    The seven convolutional layer groups are

    In the 50×46×32-sized micro-Doppler pattern feature map output through the fifth convolutional layer group, 64 filters of 3×3×32 size are moved up, down, left, and right by one padding at 2-pixel intervals in the 25×23 A first convolutional layer for extracting a micro-Doppler pattern feature map with a size of ×64, and 64 filters of a size of 3×3×32 from a micro-Doppler pattern feature map with a size of 25×23×64 output through the first convolution layer A signal using a vehicle radar, including a sixth convolutional layer group including a second convolutional layer for extracting a 25 × 23 × 64 micro Doppler pattern feature map by moving 1 padding at 1 pixel intervals in the vertical and horizontal directions. processing unit.
17. 17. The method of claim 16,

    The seven convolutional layer groups are

    In the 25×23×64 micro-Doppler pattern feature map output through the sixth convolutional layer group, 64 filters of 3×3×64 size are moved up, down, left, and right by one padding at 1-pixel intervals in the 25×23 A first convolutional layer for extracting a micro-Doppler pattern feature map with a size of ×64, and 64 filters of a size of 3×3×32 from a micro-Doppler pattern feature map with a size of 25×23×64 output through the first convolution layer A signal using a vehicle radar, including a seventh convolutional layer group including a second convolutional layer for extracting a micro-Doppler pattern feature map of 25 × 23 × 64 by moving one padding at intervals of one pixel in the vertical, left and right directions. processing unit.
18. 18. The method according to any one of claims 12 to 17,

    The second convolutional layer group to the seventh convolutional layer group include,

    A signal processing device using a vehicle radar that combines element-specific information of the micro Doppler pattern feature map extracted in the previous step and the micro Doppler pattern feature map extracted in the current step through a shortcut.
19. 18. The method of claim 17,

    The average full layer is,

    In the micro-Doppler pattern feature map of size 23×23×64 output through the seventh convolutional layer group, one filter of size 7×7×64 is moved up, down, left, and right by one padding at 1-pixel intervals in the direction of 1×1 Extracting a micro-Doppler pattern feature map of ×64 size,

    The fully connected layer is connected to the average full layer and has 64 and 2 nodes as inputs and outputs, respectively, and reduces the micro-micro Doppler pattern feature map using weights of 64×2,

    The softmax layer is a signal processing device using a vehicle radar to calculate a probability that the micro-micro Doppler pattern feature map output through the fully connected layer is a pedestrian or a non-pedestrian.
20. 20. The method of claim 19,

    The classification module is

    A signal processing apparatus using a vehicle radar for classifying the object as a pedestrian or a non-pedestrian according to the calculated probability.
21. In the RF module, the transmitted FMCW (Frequency Modulation Continuous Wave) signal and the first that is reflected by an object whose position including movement or movement with time is changed to generate a beat frequency signal that is a difference frequency between the received FMCW signal step;

    a second step of generating, in a signal processing module, an image of a micro-Doppler frequency change pattern based on the generated beat frequency signal; and

    A third step of classifying the object according to the generated image of the micro-Doppler frequency change pattern in a CNN (Convolution Neural Network) module; including; a signal processing method using a vehicle radar.
22. A computer-readable recording medium recording a program for executing the method of claim 21 on a computer.

Images