KR20220165034A - Pedestrian walking safety device and method using sound-based mobile app - Google Patents

Abstract

The present invention relates to a pedestrian safety device and method using a sound-based mobile app. The device includes: a learning algorithm loading unit for loading a learning algorithm for situational awareness based on sound data; a data collection unit for collecting ambient sound in real time based on pedestrians; a situation recognition unit for recognizing an environment necessary for safe walking of the pedestrian by learning the collected sound data through the loaded learning algorithm; and a notification providing unit for providing notification of a dangerous situation to the pedestrian based on the situation recognition result.

Description

Pedestrian walking safety device and method using sound-based mobile app {PEDESTRIAN WALKING SAFETY DEVICE AND METHOD USING SOUND-BASED MOBILE APP}

The present invention relates to a pedestrian safety technology, and more particularly, to pedestrian safety using a sound-based mobile app capable of detecting situational awareness and approaching a moving object through a pedestrian ambient sound through a dedicated app and notifying the pedestrian of a dangerous situation in advance. It relates to an apparatus and method.

Due to the emergence of various means of transportation, the road environment is becoming more complex and dangerous. In particular, the risk of safety accidents on pedestrian roads is rapidly increasing due to the increase in Personal Mobility (PM), which can be accessed on pedestrian roads, in addition to vehicles running on existing roads such as self-driving cars, electric kickboards, electric wheelchairs, and delivery vehicles/robots. The situation is.

Among traffic deaths, the proportion of pedestrian deaths is 39.9%, which is more than twice as high as the OECD average of 18.6%.

As the number of personal mobility users increases, traffic accidents are also rapidly increasing. A significant number of users mainly use bicycle roads or sidewalks, and it is important to prevent accidents in that they can cause fatal damage to not only users but also cyclists or pedestrians around them.

Korean Patent Registration No. 10-1683012 (2016.11.30)

One embodiment of the present invention is to provide a pedestrian safety device and method using a sound-based mobile app capable of detecting situational awareness and approaching a moving object through a pedestrian ambient sound through a dedicated app and notifying the pedestrian of a dangerous situation in advance.

An embodiment of the present invention is to provide a pedestrian safety device and method using a sound-based mobile app that operates in conjunction with an app running on a user terminal and can effectively perform sound collection and situational awareness for pedestrian safety. do.

An embodiment of the present invention is applied to personal mobility to provide a pedestrian safety device and method using a sound-based mobile app that can contribute to preventing safety accident risks of personal mobility accessible on a walking path.

Among embodiments, a pedestrian safety device using a sound-based mobile app includes a learning algorithm mounting unit equipped with a learning algorithm for situational awareness based on sound data, and data collecting ambient sound in real time based on a pedestrian. A collection unit, a situation recognition unit that recognizes an environment necessary for safe walking of the pedestrian by learning the collected sound data through the built-in learning algorithm, and a dangerous situation for the pedestrian based on the result of the situation recognition. A notification providing unit for providing a notification is included.

The learning algorithm loading unit may include only lite algorithms optimized for prior sound data learning through a server.

The data collection unit may sense ambient sound of the walking pedestrian through a microphone sensor, convert the sensed one-dimensional signal into two-dimensional data, and collect the same.

The microphone sensor may be included in a pedestrian terminal or implemented in the form of a wearable device.

The context recognition unit may perform a context classification operation on the collected sound data based on the learning algorithm to recognize a surrounding environment and an approaching state of a moving object that is a risk factor to pedestrians.

The context-aware unit uses the collected sound data as a model input of the learning algorithm to perform situational classification including a moving object, a signal sound, environment, and characteristics, and determines the surrounding environment of the pedestrian to determine whether the moving object is detected by the walking pedestrian. You can decide which situation to approach.

The notification providing unit may notify the pedestrian of a dangerous situation through at least one of voice output, vibration, and text notification through a speaker set by the pedestrian.

Among the embodiments, a pedestrian safety method using a sound-based mobile app includes the steps of installing a lite algorithm optimized for prior sound data learning in a pedestrian terminal, collecting the pedestrian's surrounding environment sound data in real time, the Classifying environmental sounds by operating a built-in sound data learning algorithm to perform learning using the sound data collected in real time as an input, and recognizing a dangerous situation by recognizing a moving object approach situation and surrounding environment information for a pedestrian based on the classification result , and notifying the pedestrian of the dangerous situation of the pedestrian according to a set method.

The disclosed technology may have the following effects. However, it does not mean that a specific embodiment must include all of the following effects or only the following effects, so it should not be understood that the scope of rights of the disclosed technology is limited thereby.

Pedestrian safety device and method using a sound-based mobile app according to the present invention detects situational awareness and approach of a moving object through a dedicated app with sounds around the pedestrian and informs the pedestrian of a dangerous situation in advance, thereby reducing traffic such as children and the elderly. Accidents of the weak can be prevented and safe walking can be done.

Pedestrian walking safety device and method using a sound-based mobile app according to the present invention operates in conjunction with an app running on a user terminal and can effectively perform sound collection and context awareness for pedestrian walking safety.

Pedestrian walking safety device and method using a sound-based mobile app according to the present invention can be applied to personal mobility and contribute to preventing the risk of safety accidents of personal mobility accessible on a walking road.

1 is a view illustrating a pedestrian walking safety system according to the present invention.
2 is a diagram explaining the physical configuration of the pedestrian walking safety device according to the present invention.
FIG. 3 is a diagram illustrating a functional configuration of a process in the pedestrian walking safety device of FIG. 2 .
4 is a flowchart illustrating a pedestrian walking safety method according to the present invention.
5 is a diagram illustrating a CNN model for learning according to the present invention.

Since the description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiment can be changed in various ways and can have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, the scope of the present invention should not be construed as being limited thereto.

Meanwhile, the meaning of terms described in this application should be understood as follows.

Terms such as "first" and "second" are used to distinguish one component from another, and the scope of rights should not be limited by these terms. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected to the other element, but other elements may exist in the middle. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that no intervening elements exist. Meanwhile, other expressions describing the relationship between components, such as “between” and “immediately between” or “adjacent to” and “directly adjacent to” should be interpreted similarly.

Expressions in the singular number should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as “comprise” or “having” refer to an embodied feature, number, step, operation, component, part, or these. It should be understood that it is intended to indicate that a combination exists, and does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In each step, the identification code (eg, a, b, c, etc.) is used for convenience of explanation, and the identification code does not describe the order of each step, and each step clearly follows a specific order in context. Unless otherwise specified, it may occur in a different order than specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be implemented as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices storing data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Terms defined in commonly used dictionaries should be interpreted as consistent with meanings in the context of the related art, and cannot be interpreted as having ideal or excessively formal meanings unless explicitly defined in the present application.

1 is a view illustrating a pedestrian walking safety system according to the present invention.

Referring to FIG. 1 , the pedestrian safety system 100 may include a pedestrian terminal 110 , a microphone sensor 130 , a sound data learning server 150 and a database 170 .

The pedestrian terminal 110 may correspond to a computing device capable of using the pedestrian walking safety service according to the present invention. The pedestrian terminal 110 may be implemented as a smart phone, a laptop computer, or a computer, but is not necessarily limited thereto, and may also be implemented as various devices such as a tablet PC. The pedestrian terminal 110 may be connected to the sound data learning server 150 through a network to exchange data. A pedestrian may input a sound signal as input data of a learning model for recognizing a surrounding environment through the pedestrian terminal 110 and recognize a surrounding situation related to an environmental sound output by the learning model. In one embodiment, the pedestrian terminal 110 may directly generate an ambient sound signal. For example, the pedestrian terminal 110 may include a microphone module that records ambient sounds, and through this, ambient sounds may be collected as environmental sounds. In another embodiment, the pedestrian terminal 110 may receive surrounding sound signals. For example, the pedestrian terminal 110 may be connected to the microphone sensor 130 that detects ambient sound, and through this, the ambient sound may be collected as environmental sound.

In addition, the pedestrian terminal 110 may be implemented by including the pedestrian walking safety device according to the present invention. In this case, the pedestrian safety device according to the present invention may correspond to a dedicated application executed on the pedestrian terminal 110 . That is, the pedestrian safety device according to the present invention can be implemented by being included in the pedestrian terminal 110 as an independent module that performs a predetermined function, and can be implemented by interworking with the sound data learning server 150 according to the present invention. It is possible to perform specific operations for pedestrian safety using the mobile app of .

The microphone sensor 130 may correspond to a sound sensor capable of detecting sounds existing around the pedestrian. That is, the microphone sensor 130 may perform an operation of detecting the sound of a moving object approaching the pedestrian. The microphone sensor 130 is implemented in an independent form separate from the pedestrian terminal 110 and can be worn by a user walking on a walking path or moving using a moving object. For example, the microphone sensor 130 may be implemented as a wearable device such as a neckband type or a wrist watch type, but is not limited thereto and may be implemented in a form mounted on a protective equipment for riding a moving object such as a helmet. The microphone sensor 130 is also called a sound sensor or a sound sensor, and is a sensor that collects ambient sound through a microphone and measures the volume (decibels) regardless of the height (frequency) of the sound. From the sound data collected through the microphone sensor 130, it is possible to recognize an approaching situation of a moving object on a walking path and to recognize surrounding environment information.

The sound data learning server 150 is implemented as a server corresponding to a computer or program capable of learning sound data to build a learning model for classifying environmental sounds and classifying environmental sounds based thereon to perform situational awareness of the surrounding environment. It can be. The sound data learning server 150 may be connected to the pedestrian terminal 110 through a wireless network such as Bluetooth or WiFi, and may transmit/receive data with the pedestrian terminal 110 through the network. In addition, the sound data learning server 150 may be implemented to operate in conjunction with a separate external system (not shown in FIG. 1) to collect data or provide additional functions. For example, the external system may include a big data server capable of building big data by integrating information collected from various environments, and a cloud server capable of providing various cloud services by being connected to the pedestrian terminal 110. have.

2 is a diagram explaining the physical configuration of the pedestrian walking safety device according to the present invention.

Referring to FIG. 2 , the pedestrian safety device 200 may be implemented by including a processor 210, a memory 230, a user input/output unit 250, and a network input/output unit 270.

The processor 210 may execute a procedure for processing each step in the operation of the pedestrian safety device 200, manage the memory 230 read or written throughout the process, and memory ( Synchronization time between the volatile memory and the non-volatile memory in 230) can be scheduled. The processor 210 can control the overall operation of the pedestrian safety device 200, and is electrically connected to the memory 230, the user input/output unit 250, and the network input/output unit 270 to control data flow between them. can do. The processor 210 may be implemented as a central processing unit (CPU) of the pedestrian safety device 200 .

The memory 230 is implemented as a non-volatile memory such as a solid state drive (SSD) or a hard disk drive (HDD) and may include an auxiliary memory device used to store all data necessary for the pedestrian safety device 200, , may include a main memory implemented as a volatile memory such as RAM (Random Access Memory).

The user input/output unit 250 may include an environment for receiving user input and an environment for outputting specific information to the user. For example, the user input/output unit 250 may include an input device including an adapter such as a touch pad, a touch screen, an on-screen keyboard, or a pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input/output unit 250 may correspond to a computing device connected through a remote connection, and in such a case, the pedestrian safety device 200 may be implemented as an independent server.

The network input/output unit 270 includes an environment for connecting to an external device or system through a network, and includes, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), and a VAN ( An adapter for communication such as Value Added Network) may be included.

FIG. 3 is a diagram illustrating a functional configuration of a process in the pedestrian walking safety device of FIG. 2 .

Referring to FIG. 3 , the pedestrian safety device 200 may include a learning algorithm loading unit 310, a data collection unit 330, a situation awareness unit 350, a notification providing unit 370, and a control unit 390. have. In one embodiment, each functional component constituting the pedestrian safety device 200 may be implemented by being included in the pedestrian terminal 110, and the pedestrian terminal 110 and the sound data learning server 150 may be interlocked organically. may be combined with

The learning algorithm loader 310 may load a learning algorithm for context awareness based on sound data. Here, the learning algorithm may include only the lite algorithm already optimized for sound data learning through the sound data learning server 150.

The sound data learning server 150 may generate a data set by collecting raw signals related to environmental sounds. Here, the sound data learning server 150 may directly collect various sounds of a pedestrian-oriented surrounding environment rather than a road where vehicles travel through a separate sound collection device, or may use a pre-built data set as needed. have. The data set consists of sounds generated by moving objects (buses, dump trucks, containers, ready-mixed concrete, motorcycles, sports cars, scooters, cars, garbage trucks, carts, etc.) etc.), as well as sounds (bicycle, quickboard, pedestrian, music/sound, dog barking, carrier, street performance, construction, outdoor roadside/park, indoor office/café, etc.) ), which may correspond to a set of raw signals for each surrounding environment or situation.

The sound data learning server 150 may generate a plurality of sound frames having a predetermined time length by sampling from raw signals classified according to surrounding environments or situations. The sound data learning server 150 may augment data by separating a corresponding signal into phase and magnitude components for a plurality of sound frames. Data augmentation may correspond to a very common technique in deep learning-based image processing. A major benefit of data augmentation in image processing is that the human eye can easily detect many types of image patterns. However, it can be difficult to accurately measure sound signal patterns such as frequency estimation, SNR calculation, etc. in the human ear.

In addition, data augmentation can be used for the purpose of increasing the number of training data per class in addition to generating realistic examples, and through this, a machine learning model that provides better accuracy can be built.

In one embodiment, the sound data learning server 150 may perform a Short Time Fourier Transform (STFT) to separate phase and magnitude components. The sound data learning server 150 may build a learning model for environmental sound classification by learning the augmented data. Here, the sound data learning server 150 may build a 2D CNN model receiving the STFT spectrogram as an input. The sound data learning server 150 uses the image data converted to the STFT spectrogram to classify ambient sound or spot sound after primary learning to determine CNN learning for the surrounding environment or presence or absence of moving object access. Each learning model can be built by performing CNN learning for each situation. The sound data learning server 150 may perform learning using a Visual Geometry Group (VGG) model. This will be described in more detail in FIG. 5 .

The data collection unit 330 may collect environmental sounds around the pedestrian in real time through the microphone module or the microphone sensor 130 included in the pedestrian terminal 110 . Here, the microphone sensor 130 is a smartphone carried by a pedestrian, a neckband type worn around a person's neck, a wrist watch type, an electric assistive device for the elderly or disabled, a motorcycle, a scooter, a quick board, a van, a bus, a dump truck, a container, It can be included in ready-mixed concrete, sports cars, cars/vans, garbage trucks, handcarts, etc. In one embodiment, the data collection unit 330 may sense ambient sound of a walking pedestrian through the microphone sensor 130 and convert the sensed one-dimensional signal into two-dimensional data. Here, the data collector 330 may convert the collected sound data into a Short-Time Fourier Transform (STFT) spectrogram, which is an image form having two axes of frequency and time.

In the spectrogram, the x-axis is the time axis (unit: frame), the y-axis is the frequency, and the value of each time frequency is expressed in color according to the size of the value, so that 3D is expressed in 2D. That is, each frame (signal cut in a short time) expresses how much each frequency component has, and converting the magnitude part of the STFT into a db scale becomes a spectrogram.

The situation recognition unit 350 may recognize an environment required for safe walking based on pedestrians. Here, environmental awareness required for safe walking may consist of recognizing the pedestrian's own location and recognizing the surrounding environment including dangerous objects. For this environment perception, the present invention uses sound data.

The situation recognition unit 350 may perform learning through a learning algorithm on the sound data collected in real time through the data collection unit 330 to recognize the surrounding environment and recognize the approaching situation of a moving object to a pedestrian. In one embodiment, the situation recognition unit 350 may perform learning of the STFT spectrogram according to a learning algorithm loaded in the learning algorithm loading unit 310 to recognize the surrounding environment and determine the proximity of a moving object to a pedestrian. . In an embodiment, the context awareness unit 350 may perform a classification operation on a given environmental sound based on a learning model of a learning algorithm. Here, the context-aware unit 350 may selectively apply one of the relevant models according to the environment in which the classification operation is performed when a CNN model for the surrounding environment and a CNN model for each context according to the presence or absence of moving objects are built as a learning model. have. The context-aware unit 350 can quickly process classification calculations by learning sound data converted into images through a learning model, and can generate classification results with higher accuracy. The situation recognition unit 350 may recognize surrounding environment information of the pedestrian based on the environmental sound classification result, and may determine the proximity of the moving object to the pedestrian. Here, the surrounding environment information may include conditions of an indoor environment, such as a quiet office or a noisy cafe, and an outdoor environment, such as a quiet park or a noisy city.

In an embodiment, the situation recognition unit 350 may determine a dangerous situation for the pedestrian by integrating the surrounding environment and the situation according to whether or not the moving object is approaching as a result of the environmental sound. For example, the situation recognition unit 350 may determine that a situation in which a pedestrian is located in a safe indoor environment with a low risk of collision with a moving object when the ambient environment is classified as a sound according to an indoor environment even if the sound is classified according to a moving object approaching situation. . In one embodiment, the situation recognition unit 350 collects ambient sound based on the user of the personal mobility driving on the walking path and classifies the environmental sound through learning. It is possible to determine the risk of collision between the vehicle and the pedestrian.

The notification provision unit 370 may provide notification of a dangerous situation according to the approach of the moving object to the pedestrian according to the situation awareness result of the situation recognition unit 350 . That is, the notification providing unit 370 may visually or audibly provide an accident risk according to the approach of a moving object in a manner set in the pedestrian terminal 110 . For example, the notification providing unit 370 may output a voice through a speaker module of the pedestrian terminal 110 or notify the pedestrian of a dangerous situation through vibration or text notification so that an accident can be prevented.

In one embodiment, the notification providing unit 370 may provide a notification of a collision risk situation with a moving object or a pedestrian to a personal mobility user driving on a walking path. That is, the notification providing unit 370 may transmit a warning alarm or text notification to the personal mobility user's terminal or personal mobility.

The control unit 390 controls the overall operation of the pedestrian safety device 150, and controls flow or You can manage data flow.

4 is a flowchart illustrating a pedestrian walking safety method using a sound-based mobile app according to the present invention.

Referring to FIG. 4 , the pedestrian safety device 200 may load the lite algorithm optimized for sound data learning in the sound data learning server 150 into the pedestrian terminal 110 (step S410). The pedestrian safety device 200 may collect environmental sound data around the pedestrian in real time through a microphone module that records ambient sound of the pedestrian terminal 110 (step S430).

In addition, the pedestrian safety device 200 operates a sound data learning algorithm mounted in the pedestrian terminal 110 to perform learning using sound data collected in real time as an input, and adaptively processes environmental sound classification to improve the surrounding environment. It can be recognized and the approaching situation of the moving object to the pedestrian can be determined (step S450). The pedestrian safety device 200 may notify the pedestrian of a pedestrian's dangerous situation through a speaker module of the pedestrian terminal 110 through a voice output or a warning alarm, or by various methods such as notification talk (step S470).

5 is a diagram illustrating a CNN model for learning sound data according to an exemplary embodiment.

Referring to FIG. 5 , the pedestrian safety device 200 may learn sound data and use a Visual Geometry Group (VGG) model as a learning model for classifying environmental sounds for each situation. There are four CNN architectures: AlexNet, VGG, GoogLeNet, and ResNet.

The VGG model fixes the input image size to 224×224, and uses 3,3 and 1 for all convolutions and a stride of 1. That is, convolution may be performed along the time and frequency axes with the STFT spectrogram as an input. Each convolution block may include convolution, activation, and batch normalization layers. In the next layer within each convolution block, the 'ReLu' activation function can be used to provide non-linearity. In the case of the last layer, a softmax activation function may be used to provide prediction probabilities according to the number of output classes.

The pedestrian safety device 200 may learn the characteristics of sound using the VGG model and build a learning model capable of classifying sounds into categories for each situation according to surrounding environment information and whether or not a moving object is approaching.

Therefore, the present invention enables quick and accurate situation recognition with little learning of environmental sound data collected in real time through a pre-learned model, and enables pedestrians to walk safely through warning alarms for dangerous situations, thereby enabling pedestrians to walk while walking on a walking path. possible accidents can be prevented.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: Pedestrian Pedestrian Safety System
110: pedestrian terminal 130: microphone sensor
150: sound data learning server
200: pedestrian safety device
210: processor 230: memory
250: user input/output unit 270: network input/output unit
310: learning algorithm loading unit 330: data collection unit
350: situation awareness unit 370: notification providing unit
390: control unit

Claims (8)

a learning algorithm loading unit for loading a learning algorithm for situational awareness based on sound data;
a data collection unit that collects ambient sound around the pedestrian in real time;
a context recognition unit that recognizes an environment necessary for safe walking of the pedestrian by performing learning on the collected sound data through the built-in learning algorithm; and
Pedestrian safety device using a sound-based mobile app including a notification providing unit for providing notification of a dangerous situation to the pedestrian based on the situation recognition result.
The method of claim 1, wherein the learning algorithm loading unit
Pedestrian safety device using a sound-based mobile app, characterized in that it includes only lite algorithms optimized for pre-sound data learning through a server.
The method of claim 1, wherein the data collection unit
A pedestrian safety device using a sound-based mobile app, characterized in that for detecting ambient sound of the walking pedestrian through a microphone sensor and converting the detected one-dimensional signal into two-dimensional data for collection.
The method of claim 3, wherein the microphone sensor
A pedestrian safety device using a sound-based mobile app, characterized in that it is included in a pedestrian terminal or implemented in the form of a wearable device.
The method of claim 1, wherein the context awareness unit
Based on the learning algorithm, a situation classification operation is performed on the collected sound data to recognize the surrounding environment and to recognize the approaching situation of a moving object that is a risk factor to the pedestrian. Pedestrian Safety Device.
The method of claim 5, wherein the context awareness unit
The collected sound data is used as a model input of the learning algorithm to perform situational classification including moving objects, signal sounds, environments, and characteristics, and to determine a situation in which the moving object approaches the walking pedestrian by determining the surrounding environment of the pedestrian. Pedestrian safety device using a sound-based mobile app, characterized in that for determining.
The method of claim 1, wherein the notification providing unit
A pedestrian safety device using a sound-based mobile app, characterized in that for notifying the pedestrian of a dangerous situation in at least one of voice output, vibration or text notification through a speaker set by the pedestrian.
Loading the lite algorithm optimized by prior sound data learning into the pedestrian's terminal;
collecting ambient environment sound data of the pedestrian in real time;
By operating the built-in sound data learning algorithm, learning is performed using the sound data collected in real time as an input to classify environmental sounds, and recognizing the approaching situation of a pedestrian and surrounding environment information for a pedestrian based on the classification result to determine a dangerous situation step; and
Pedestrian safety method using a sound-based mobile app, including the step of notifying the pedestrian of a pedestrian's dangerous situation according to a set method.

Images