CN118265994A

CN118265994A - Apparatus and method for estimating and managing road surface type using sound signal

Info

Publication number: CN118265994A
Application number: CN202280076362.1A
Authority: CN
Inventors: 金玟贤
Original assignee: Mobayei Co ltd
Current assignee: Mobayei Co ltd
Priority date: 2021-11-17
Filing date: 2022-11-11
Publication date: 2024-06-28

Abstract

According to various embodiments, an electronic device for classifying a road surface using sound signals and a road surface classification method using the same are disclosed. Further, an apparatus and method for managing a road surface by road surface classification are disclosed. Meanwhile, an installation method of an infrastructure for implementing the road surface classification method is disclosed. Other embodiments are also possible.

Description

Apparatus and method for estimating and managing road surface type using sound signal

Technical Field

The present disclosure relates to an apparatus for estimating a road surface type using sound signals and a method of classifying and managing a road surface using the same, and more particularly, to an apparatus for classifying sound signals reflected on a road surface using an artificial neural network and controlling a road surface or a moving object based on the classified road surface and a method of managing a road surface using the same.

Background

In general, since a moving object moving on the ground performs acceleration and deceleration control according to its moving ground plane (i.e., according to the friction coefficient of the road surface), accurate estimation of the friction coefficient of the road surface is very important for stable control and maximum motion performance control.

The black ice accident in which winter is drastically increased occurs due to the drastic change of the road surface friction coefficient without being recognized, and thus, it can be said that the demand for the friction coefficient estimation technique is fully embodied.

In addition, the regenerative braking technology is critical to improve energy efficiency of recently commercialized electric vehicles, and the demand for a technology of estimating the road surface friction coefficient is expanding in terms of ensuring running stability when regenerative braking is applied.

As a method for estimating such a road surface state or estimating a road surface friction coefficient based on such a road surface state, in the related art, a method using dynamic information of a vehicle and a method using sensed information are mainly used.

In the case of a method using dynamic information of a vehicle, measurement information of various sensors mounted on the vehicle is substituted into a vehicle dynamic model and estimated. In this case, there are disadvantages in that accuracy is lowered in the case of deviation from a predetermined model, and since measurement can be performed only after passing the road surface, the friction coefficient of the road surface cannot be estimated.

Further, in the case of the electromagnetic wave sensor-based method (e.g., image information), the friction coefficient of the road surface can be estimated remotely, but there are limitations: the result may vary depending on the mounting position or direction of the sensor, and an expensive sensor device and a signal processing apparatus for the sensor device are required.

Meanwhile, a road surface estimation technique using acoustic information has been actively discussed, and in the related art, a technique of estimating a road surface state based on frictional sound between the ground and a tire has been emphasized, which has limitations in that accuracy is insufficient and a front road surface state cannot be checked before passing the road surface.

Therefore, in the conventional method, the road surface state cannot be predetermined, or the determination process is uneconomical and inaccurate, and there is a limitation in that the above-described problem cannot be effectively solved.

Disclosure of Invention

[ Technical problem ]

In order to solve the above-described problems, the present disclosure is directed to an apparatus and method for estimating a road surface type using sound signals.

Further, the present disclosure aims to provide an apparatus and method for controlling and managing a road surface in real time by estimating the type of the road surface according to the present disclosure.

Meanwhile, technical problems to be solved by the present disclosure are not limited to the above technical problems, and technical problems not mentioned can be clearly understood from the present specification and drawings by those skilled in the art to which the disclosure included in the present disclosure pertains.

[ Technical solution ]

According to embodiments of the present disclosure, an electronic device for classifying a road surface using sound signals includes: a transceiver configured to transmit and receive sound signals; an atmospheric sensor; and at least one processor electrically connected to the transceiver and the atmospheric sensor, wherein the at least one processor is configured to: transmitting, using the transceiver, an acoustic signal to a target pavement spaced a first distance from the electronic device; receiving a reflected signal of the sound signal of the target road surface using the transceiver; acquiring atmospheric information related to the sound signal using the atmospheric sensor; acquiring first data of the received reflected signal; generating second data by correcting the first data based on the atmospheric information; acquiring third data related to frequency domain information of the second data based on the second data; and determining a type of the target road surface based on the third data and a road surface classification artificial neural network, and wherein the road surface classification artificial neural network is trained to a frequency domain data set generated based on sound signals reflected from a road surface at a second distance different from the first distance.

According to an embodiment of the present disclosure, the second data may be generated by correcting the first data based on the atmospheric information and the first distance.

Further, the first distance may be estimated based on a transmission time of the sound signal and a reception time of the reflected signal.

According to an embodiment of the present disclosure, the third data may be acquired by converting the second data into STFT (short time fourier transform).

According to an embodiment of the present disclosure, the at least one processor may be configured to generate a signal for controlling a road surface management device mounted on the target road surface based on the determined type of the target road surface.

According to embodiments of the present disclosure, the pavement management apparatus may include a heating wire apparatus or a brine spray apparatus.

According to an embodiment of the present disclosure, the at least one processor may be configured to determine whether a preset weather condition is satisfied, and generate a signal for controlling the road surface management device when the preset weather condition is satisfied.

Further, the at least one processor may be configured to determine whether the type of the target road surface determined at a first time changes at a second time, and determine whether to generate a signal for controlling the device mounted on the target road surface based on the type of the target road surface determined at a third time when the first category determined as the type of the target road surface at the first time is different from the second category determined as the type of the target road surface at the second time.

According to an embodiment of the present disclosure, the type of the target road surface may be determined at each first period, and the at least one processor may be configured to determine the type of the target road surface at each second period when the type of the target road surface is determined as the first category.

According to an embodiment of the present disclosure, the electronic device may further include at least one of an infrared sensor for acquiring temperature information of the target road surface or a visual sensor for acquiring image information of the target road surface, and the at least one processor may be configured to determine the type of the target road surface further based on the temperature information or the image information.

According to embodiments of the present disclosure, a method for classifying a road surface using sound signals performed by an electronic device includes: transmitting an acoustic signal to a target road surface spaced a first distance from the electronic device; receiving a reflected signal of the sound signal of the target road surface; acquiring atmospheric information related to the sound signal; acquiring first data of the received reflected signal; generating second data by correcting the first data based on the atmospheric information; acquiring third data related to frequency domain information of the second data based on the second data; and determining a type of the target road surface based on the third data and a road surface classification artificial neural network, and wherein the road surface classification artificial neural network may be trained to a frequency domain data set generated based on sound signals reflected from a road surface at a second distance different from the first distance.

[ Advantageous effects ]

According to the present disclosure, by rapidly and accurately classifying road types based on ultrasonic signals, road surface control or vehicle control according to the classified road surface can prevent accidents from occurring.

Further, according to the present disclosure, by automatically controlling the road surface management using the road surface classification information, the road surface can be managed economically and efficiently.

In addition, the method and the system can provide more effective traffic network information for users by acquiring the road surface information in real time.

Meanwhile, the effects of the present disclosure are not limited to the above-described effects, and the effects not mentioned may be clearly understood by those skilled in the art to which the present disclosure pertains from the specification and drawings.

Drawings

Fig. 1 is a block diagram of a road surface classification device according to various embodiments of the present disclosure.

Fig. 2 is a diagram illustrating installation and operation of a road surface classification device in a road infrastructure according to an embodiment of the present disclosure.

Fig. 3 is a flowchart illustrating a method performed by the road surface classification apparatus according to the present disclosure.

Fig. 4 is a diagram showing sound signals emitted by the road surface classification apparatus according to the embodiments of the present disclosure on a time axis.

Fig. 5 is a diagram illustrating a transmission interval of a sound signal and a reception interval of a reflected signal according to an embodiment of the present disclosure.

Fig. 6 is a diagram illustrating an object mounted with a road surface classification apparatus according to various embodiments of the present disclosure.

Fig. 7 is a diagram illustrating a method of acquiring a data set for learning a road surface classification artificial neural network according to embodiments of the present disclosure.

Fig. 8 is a flowchart illustrating a process of preprocessing a reflected signal received by a road surface classification device according to embodiments of the present disclosure.

Fig. 9 is a diagram illustrating a multi-modal artificial neural network according to an embodiment of the present disclosure.

Fig. 10 is a flowchart showing an operation of the road surface classification apparatus according to an embodiment of the present disclosure to change a control operation based on a predetermined control change trigger.

Fig. 11 is a diagram illustrating a scene in which a road surface classification result is changed according to an embodiment of the present disclosure.

Fig. 12 is a diagram illustrating a road surface management method of the road surface classification apparatus according to various embodiments of the present disclosure.

Fig. 13 is a diagram illustrating a road surface classification apparatus according to an embodiment of the present disclosure collecting traffic information.

Fig. 14 is a configuration diagram of a road surface type estimating apparatus according to an embodiment of the present disclosure.

Fig. 15 is a diagram for explaining a transmission signal and a reception signal in a road surface type estimation device using sound according to an embodiment of the present disclosure.

Fig. 16 is a diagram for exemplarily explaining a signal converter in a road surface type estimation apparatus using sound according to an embodiment of the present disclosure.

Fig. 17 is a diagram for explaining an artificial neural network in a road surface type estimating apparatus using sound according to an embodiment of the present disclosure.

Fig. 18 is a diagram for explaining the operation of the convolution execution unit.

Fig. 19 is a diagram for explaining a code of a convolution execution unit of the road surface type estimation device using sound of the present disclosure.

Fig. 20 is a flowchart of a road surface type estimation method using domain transformation of sound according to an embodiment of the present disclosure.

Fig. 21 is a flowchart illustrating an embodiment of a method of estimating a road surface type using sound according to the present disclosure.

Fig. 22 is a flowchart illustrating a method of estimating a road surface type using an atmospheric attenuation corrected sound according to an embodiment of the present disclosure.

Fig. 23 is a diagram for explaining a road condition monitoring system equipped with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 24 is a construction diagram of a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 25 is a diagram showing an example of identifying a uniform road surface state in a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 26 is a diagram showing an example of identifying an uneven road surface condition in a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 27 is a diagram for explaining a method of locating a divided area of a sound sensor in a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 28 is a diagram for explaining an example of an artificial neural network of a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 29 is a diagram for explaining an example of a division processing unit of a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 30 is a flowchart of a monitoring method in a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 31 is a detailed flowchart of step 3050 of fig. 30 for analysis by merging.

Fig. 32 is a construction diagram of a road condition monitoring system provided with a visual sensor and a sound sensor according to another embodiment of the present disclosure.

Fig. 33 is a flowchart of another embodiment of a monitoring method in a road condition monitoring system provided with a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Fig. 34 is a diagram for explaining the operation of the control system of the road heating wire device according to an embodiment of the present disclosure.

Fig. 35 is a configuration diagram of a control system of a road freeze-proofing device according to an embodiment of the present disclosure.

Fig. 36 is a configuration diagram of a control system of a road freeze protection device according to another embodiment of the present disclosure.

Fig. 37A to 37C are diagrams for explaining an artificial intelligence analysis model used in a control system of a road freeze-proofing device according to an embodiment of the present disclosure.

Fig. 38 is a flowchart of a control method of the road freeze protection device according to an embodiment of the present disclosure.

Fig. 39 is a detailed flowchart of the control signal generating step 3850 of fig. 38 when the road freeze-protection device is a heater wire device according to an embodiment of the present disclosure.

Fig. 40 is a detailed flowchart of the control signal generating step (3850) of fig. 38 when the road freeze protection device is a brine spray device, according to an embodiment of the disclosure.

Fig. 41 is a perspective view schematically showing a road infrastructure sensor configuration structure according to an embodiment of the present disclosure.

Fig. 42 is a side view schematically illustrating a road infrastructure sensor construction structure according to an embodiment of the present disclosure.

Fig. 43 is a schematic perspective view (a) and a partial cross-sectional perspective view (b) showing a sound sensor unit according to an embodiment of the present disclosure.

Fig. 44 is a side sectional view schematically illustrating a sound sensor unit according to an embodiment of the present disclosure.

Fig. 45 is a perspective view schematically showing a road infrastructure sensor configuration structure according to other embodiments of the present disclosure.

Fig. 46 is a side view schematically illustrating a road infrastructure sensor construction structure according to other embodiments of the present disclosure.

Fig. 47 is a partially enlarged perspective view (a), a bottom perspective view (b), and a partial cross-sectional perspective view (b) schematically illustrating a road infrastructure sensor construction structure according to other embodiments of the present disclosure.

Fig. 48 is a partial cross-sectional side view schematically illustrating a sound sensor unit according to other embodiments of the present disclosure.

Fig. 49 is a flowchart showing a construction method of the road infrastructure sensor construction structure according to the preferred embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In describing the embodiments, description of technical contents known in the art to which the present disclosure pertains and not directly related to the present disclosure will be omitted. This is to clearly communicate the subject matter of the present disclosure by omitting unnecessary descriptions.

For the same reason, some components are enlarged, omitted, or schematically shown in the drawings. In addition, the dimensions of the individual components do not fully reflect the actual dimensions. In each drawing, identical or corresponding parts have identical reference numerals.

The advantages and features of the present disclosure and methods of accomplishing the same may become apparent from the following detailed description of embodiments taken in conjunction with the accompanying drawings. It should be understood, however, that the disclosure from the drawings does not specify or limit the embodiments and includes all modifications, equivalents, and alternatives falling within the spirit and technical scope of the embodiments. The specific structural or functional descriptions of the various embodiments are presented for purposes of illustration only and the embodiments of the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments explicitly described in the specification or the present application.

That is, the embodiments of the present disclosure are provided so that the present disclosure is comprehensive and provides a given scope of the present disclosure to one of ordinary skill in the art to which the present disclosure pertains, and the invention of the present disclosure is defined only by the scope of the claims. Like reference numerals refer to like parts throughout the specification.

Terms such as "first" and/or "second" may be used to describe various components, but the components should not be limited by these terms. These terms are only used to distinguish one element from another element, e.g., a first element may be termed a second element, and, similarly, a second element may be termed a first element, without departing from the scope of the present concepts.

It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element but other elements may be present therebetween. On the other hand, when an element is referred to as being "directly connected" or "directly coupled" to another element, it is understood that there are no other elements in between. Other expressions describing the relationship between components, i.e. "between" and "directly between" or "adjacent" and "directly adjacent" should be interpreted as well.

In the drawings, each block of the process flow diagrams and combinations of flow diagrams can be implemented by computer program instructions. Because such computer program instructions may be installed in a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions described in the flowchart block(s). Because such computer program instructions may be stored in a computer-usable or computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, the instructions stored in the computer-usable or computer-readable memory produce an article of manufacture including instruction means that implement the function specified in the flowchart block(s). Because the computer program instructions may be loaded onto a computer or other programmable data processing apparatus, the instructions which execute the functions specified in the flowchart block(s) are provided in the computer or other programmable data processing apparatus by executing a sequence of operational steps on the computer or other programmable data processing apparatus.

Further, each block may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). Furthermore, it should be noted that in some alternative implementations, the functions noted in the block may occur out of the order. For example, two blocks shown in succession may be executed substantially concurrently or the blocks may be executed in the reverse order, depending upon the functionality involved.

The term "unit" as used in this disclosure refers to a software or hardware component, such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC). "units" play a particular role but are not limited to software or hardware. The "unit" may be constructed in an addressable storage medium or may be configured to reproduce one or more processors. Thus, according to some embodiments, a "unit" includes components, such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided for in the components and "units" may be combined into a fewer number of components and "units" or may be further separated into additional components and "units. Furthermore, the components and "units" may be implemented as one or more CPUs in a reproduction device or a secure multimedia card. Further, according to embodiments of the present disclosure, a "unit" may include one or more processors.

The operating principle of the present disclosure will be described in detail below with reference to the accompanying drawings. In the following description of the present disclosure, a detailed description of related known functions or constructions will be omitted when it may unnecessarily obscure the subject matter of the present disclosure. Further, the terms described below are terms defined in consideration of functions in the present disclosure, which may be changed according to intention or habit of a user or operator. Therefore, the definition should be made according to the content of the entire specification.

The present disclosure relates to a system for classifying a road surface using sound signals and thereby managing the operation of the road surface or vehicle.

The road surface classifying device according to an embodiment of the present disclosure may include a device installed in a road infrastructure or a moving object and determining the type or state of the road surface.

The road surface classification device according to another embodiment of the present disclosure may include a server device that determines the type or state of the road surface based on information received from a device installed in a road infrastructure or a moving object.

Referring to fig. 1, a road surface classification device 100 according to an embodiment of the present disclosure may include a transceiver unit 110, a sensing unit 120, and a controller 130. Meanwhile, the road surface classifying device according to the embodiments of the present disclosure may include additional components in addition to the above-described hardware components, and is not limited to the components shown in fig. 1. Fig. 1 is a diagram for explaining hardware components constituting a road surface classification apparatus 100 of the present disclosure, and a road surface classification apparatus according to another embodiment of the present disclosure may be constructed by omitting some of the components shown in fig. 1.

The transceiver unit 110 is a hardware component configured to transmit and receive sound signals, and may include a transmitter (not shown), a receiver (not shown), or a transceiver (not shown). Hereinafter, each of the transmitter and the receiver constituting the transceiver unit 110 will be described in detail.

The transmitter is a device that generates and emits a sound signal, and may be arranged in a direction of emitting the sound signal toward the road surface. In this case, the emitted sound signal may include a high-frequency ultrasonic signal.

Meanwhile, the frequency of the generated sound signal may be fixed according to the type of the transmitter, and may be set or changed by a user input. Further, the sound signal may be transmitted through a user input, a control of a controller or a server, or the sound signal may be transmitted according to a predetermined rule, or one or more signals may be transmitted periodically within one period. In this case, the number of emitted sounds or the emission period may be variable.

The receiver is a device that receives sound signals reflected from the road surface and may be arranged to receive sound signals reflected from the road surface.

On the other hand, the receiver may directly receive sound signals transmitted from adjacent transmitters in addition to the reflected signals. The signal directly received from the transmitter is a signal that is independent of the classification of the road surface to be determined by the road surface classification device of the present disclosure, and such a noise signal may be referred to as crosstalk.

According to an embodiment of the present disclosure, the transmitter and the receiver may be mounted at a spaced apart to reduce the occurrence of crosstalk. Furthermore, according to another embodiment of the present disclosure, structures may be additionally arranged between the transmitter and the receiver to reduce the occurrence of an interference signal (e.g., crosstalk). The structure may be formed of a material or structure having physical characteristics that attenuate or absorb sound signals, and may be an electronic device configured to achieve such physical characteristics.

On the other hand, in the road surface classification device of the present disclosure, the transmitter and the receiver do not have to be physically distinguished, but may be implemented in one integrated form, for example as a transceiver. In the following description, a transceiver is a term including both a transmitter, a receiver, or a transceiver, and may refer to a hardware device in which a transmitter and a receiver are integrated together, and may refer to a device including both a physically distinct transmitter and receiver, or each of them.

The transceiver may transmit or receive sound over a range of azimuth angles depending on hardware performance. The road surface classifying device 100 according to embodiments of the present disclosure may use transceivers having different azimuth angles in consideration of the object or environment in which the road surface classifying device is installed. For example, the azimuth of the transceiver used in the road surface classification device arranged in the road infrastructure may be smaller than the azimuth of the transceiver used in the road surface classification device mounted in the vehicle.

When the transmitter and the receiver are configured to be distinguished from each other in the road surface classification device, the transmitter and the receiver according to embodiments of the present disclosure may be arranged in consideration of azimuth angles therebetween. A receiver according to an embodiment of the present disclosure may be arranged outside the azimuth range of the transmitter, and in this way, the outermost sound signal of the azimuth radiated by the transmitter may not be sensed in the receiver. A receiver according to another embodiment of the present disclosure may be arranged outside based on a center of an azimuth angle of a transmitter such that a crosstalk signal sensed by the receiver is less than or equal to a reference value.

On the other hand, the transceiver according to embodiments of the present disclosure may be designed or arranged to react only to a specific frequency characteristic of the reflected wave of the road surface.

The sensing unit 120 is a hardware component that acquires information required for road surface classification according to the present disclosure by measurement, and the sensing unit 120 according to the present disclosure may include an atmospheric sensor, a camera, and/or an infrared sensor.

The sensing unit 120 according to embodiments of the present disclosure may include an atmospheric sensor. The atmospheric sensor is a hardware device for acquiring information related to an atmospheric condition, and the atmospheric information measured or acquired by the atmospheric sensor may include at least one of temperature, humidity, or atmospheric pressure. In addition, the atmospheric information may also include information about wind. In this case, the information about the wind may include a physical quantity related to the wind, such as a wind speed, a wind quantity, or a wind direction. In the present specification, the atmospheric sensor may refer to a device including at least one of a temperature sensor, a humidity sensor, or an atmospheric pressure sensor. Further, an atmospheric sensor may refer to a device capable of sensing a plurality of different atmospheric information. An atmospheric sensor according to an embodiment of the present disclosure may measure temperature, humidity, barometric pressure, and/or wind speed at a location of a road surface classification device.

The sensing unit 120 according to embodiments of the present disclosure may further include a camera and/or an infrared sensor. The camera is a device that acquires an image, and can acquire image information about a road surface, and the infrared sensor can acquire temperature information about the road surface by detecting radiant heat emitted from the road surface. Since the temperature information acquired by the infrared sensor is temperature information about the road surface and the temperature information acquired by the atmospheric sensor is temperature information about the atmosphere, the values represented by the respective temperature information acquired by the different sensors may be different.

Various information measured or acquired by the sensing unit 120 of the present disclosure may be used in combination to improve the accuracy of road classification. That is, the road surface classification result output by the road surface classification device 100 according to the embodiments of the present disclosure may be generated based on a plurality of pieces of information, and specific embodiments thereof will be described below.

On the other hand, according to an embodiment of the present disclosure, the camera and/or the infrared sensor included in the sensing unit 120 are exemplary, and the sensing unit 120 may include any sensing device that acquires information usable to classify the road surface, in addition to the above-described atmospheric sensor, camera, or infrared sensor.

The controller 130 is configured to perform the method performed by the road surface classification device of the present disclosure, and may include at least one processor having logic and arithmetic circuitry. The controller 130 may process data according to a program and/or instructions provided from a memory (not shown) and generate a control signal according to the result of the processing.

According to various embodiments, the controller 130 may control at least one other component (e.g., hardware or software component) of the road surface classification device 100 connected to the controller 130, and may perform various data processing or arithmetic operations. According to an embodiment, the controller 130 may store a command or data received from another component (e.g., the receiver 120 or the sensing unit 120) in a volatile memory (not shown), process the command or data stored in the volatile memory (not shown), and store the resultant data in a non-volatile memory (not shown) as at least a part of data processing or arithmetic operations. For example, the signal acquired through the receiver 120 may be converted into a digital signal by an analog-to-digital converter (ADC) circuit included in the controller 130 and processed. In addition, the converted digital signal may be preprocessed as input data to be input to the artificial neural network. A specific method of processing the received signal and/or data of the present disclosure will be described below.

According to an embodiment, the controller 130 may include a main processor (e.g., a Central Processing Unit (CPU) or an Application Processor (AP)) or an auxiliary processor (e.g., a Graphics Processing Unit (GPU), a Neural Processing Unit (NPU), an image signal processor, a sensor hub processor, or a communication processor), which may operate independently or together with the main processor. For example, when the road surface classification device 100 includes a main processor and an auxiliary processor, the auxiliary processor may be set to use lower power than the main processor or be dedicated to a particular function. The secondary processor may be implemented separately from the primary processor or as part of the primary processor.

The road surface classification device according to embodiments of the present disclosure may include a communication unit (not shown). The communication unit refers to a hardware component that receives a command or data input from a user or other external device, transmits a command or data generated by the road surface classification device to the outside, or transmits a command or receives a command from other components of the road surface classification device, and may include wired and wireless communication modules and/or input/output interfaces. The road surface classification device according to an embodiment of the present disclosure may receive information from an external electronic device (e.g., a controller or a management server installed outside the road surface classification device) or transmit information generated by the road surface classification device to the external electronic device. On the other hand, the communication unit may be implemented solely by circuit elements included in the controller. That is, the road surface classification device according to the embodiments of the present disclosure may be a device that provides information necessary to classify a road surface in conjunction with an external electronic device.

According to an embodiment of the present disclosure, an artificial neural network (not shown) for classifying a road surface of the present disclosure may be included as a Software On Chip (SOC) or a Micro Controller Unit (MCU) and provided in the controller 130. Alternatively, the artificial neural network may be provided in the form of software operated by the controller 130 and updated from an external server or user input through the communication unit.

On the other hand, the artificial neural network according to the embodiments of the present disclosure may be implemented in an external electronic device (e.g., a controller or a server), and in this case, data generated based on sound signals and data (e.g., atmospheric information) required for road surface classification may be transmitted from the controller 130 of the road surface classification device to the external electronic device, and the external electronic device may classify the road surface based on the data received from the road surface classification device.

According to another embodiment of the present disclosure, the road surface classification device may be a server device. In this case, the road surface classification device may not include the transceiver unit 110 and the sensing unit 120, and may receive data required for road surface classification from an external electronic device through a communication unit (not shown), and may classify the road surface through the controller 130 based on the received data. In addition, the classified road surface classification result and/or control information related thereto may be transmitted to an external electronic device.

Referring to fig. 2, the road surface classification apparatus 100 may be installed to face a road surface 230 to be classified by the road infrastructure 200.

In the present disclosure, the road infrastructure 200 is a general term of a traffic facility including a columnar structure 210 such as a signal lamp, a street lamp, a road guidance sign, or an image information processing device installed on a road or a roadside, and refers to a structure in which a road surface classification device may be installed on a road, and is not limited to the above example. According to an embodiment of the present disclosure, the fact that the road surface classification device 100 is installed in the road infrastructure 200 may mean that it is installed at the upper end of the pillar structure 210, but is not limited thereto.

The road infrastructure 200 may include a controller 220 for controlling the electronic devices installed in the columnar structure 210. The electronic devices mounted on the columnar structure 210 may include a lighting device for a street lamp or a signal lamp, cctv, a traffic information acquisition camera, or a road surface classification device of the present disclosure.

The controller 220 is a device that controls an electronic device mounted on the pillar structure 210, and for example, when the pillar structure is a street lamp, the controller 220 may be a street lamp controller that controls an operation of the street lamp, and when the pillar structure is a signal lamp, the controller 220 may be a traffic signal controller that controls a signal of the signal lamp.

The controller 220 according to embodiments of the present disclosure may control the operation of the road surface classification apparatus 100 of the present disclosure, and may control the road surface on which the road infrastructure 200 is located based on the road surface classification information or commands obtained from the road surface classification apparatus 100.

The controller 220 according to embodiments of the present disclosure may serve as a gateway between the road surface classification device 100 and a management server (not shown). That is, the controller 220 may include a wired/wireless communication module and transmit information obtained from the road surface classification device to the management server or receive a command or data for controlling the road surface classification device or the road from the management server.

The controller 220 according to embodiments of the present disclosure may include an artificial neural network of the present disclosure, and by the artificial neural network, the road surface may be classified directly based on information acquired from the road surface classification device. In this case, since the performance of the processor or the memory included in the controller 220 may be superior to that of the processor or the memory of the road surface classification device, the artificial neural network provided to the controller 220 may be superior to that of the artificial neural network provided to the road surface classification device 100.

The controller 220 according to embodiments of the present disclosure may control the road surface management device 250 equipped in the road based on the classified road surface to manage the road surface.

The pavement management apparatus 250 according to embodiments of the present disclosure may include a snow removing apparatus (e.g., a heating wire apparatus or a brine spraying apparatus) or a drainage facility installed on a road. Details of the operation of the road surface management device according to the embodiments of the present disclosure will be described below.

Hereinafter, a method of classifying a road surface by the road surface classifying device according to the present disclosure will be described in detail.

In general, since different materials have different acoustic impedances, the reflected signal for each material is different for the same sound incident signal. Thus, by using these physical properties, the material can be distinguished by analyzing the reflected signal. In particular, since acoustic impedance is a physical quantity having frequency characteristics, if a reflected signal is analyzed in the frequency domain, the material of the reflection surface can be classified more finely.

According to various embodiments of the present disclosure, an artificial neural network may be used to perform a road surface classification method using sound reflection signals.

The neural network model of the artificial neural network according to embodiments of the present disclosure may include multiple hierarchies or layers.

The neural network model may be implemented in the form of a classifier that generates road classification information. The classifier may perform multiple classifications. For example, the neural network model may be a multiple classification model that classifies the results of input data into a plurality of categories.

A neural network model according to an embodiment of the present disclosure may include a Deep Neural Network (DNN) of a multi-layer perceptron algorithm that includes an input layer, a plurality of hidden layers, and an output layer.

A neural network model according to another embodiment of the present disclosure may include a Convolutional Neural Network (CNN). As the CNN structure, at least one of AlexNet、LENET、NIN、VGGNet、ResNet、WideResnet、GoogleNet、FractaNet、DenseNet、FitNet、RitResNet、HighwayNet、MobileNet、DeeplySupervisedNet can be used. The neural network model may be implemented using multiple CNN structures.

For example, the neural network model may be implemented to include a plurality VGGNet of blocks. As a more specific example, the neural network model may be provided by combining a first structure in which a CNN layer, a Bulk Normalization (BN) layer, and a ReLU layer having 64 filters of 3×3 size are sequentially combined, and a second block in which a CNN layer, a ReLU layer, and a BN layer having 128 filters of 3×3 size are sequentially combined.

The neural network model may include a max pooling layer after each CNN block, and may include a Global Average Pooling (GAP) layer, a Full Connectivity (FC) layer, and an activation layer (e.g., sigmoid, soft max, etc.) at the end.

An artificial neural network according to embodiments of the present disclosure refers to a neural network model for extracting features from frequency converted signals of sound signals to classify a road surface, and is not limited to the above examples.

The road surface classification artificial neural network according to the embodiments of the present disclosure may be trained by using frequency domain data of the reflected signal as an input value, and the trained artificial neural network may classify the road surface of the reflected target signal by using frequency domain data of the target signal as an input value.

The frequency domain data may refer to data obtained by performing frequency domain transformation on a digital signal converted through ADC sampling of a reflected signal.

As the frequency domain transform method according to the embodiments of the present disclosure, short Time Fourier Transform (STFT), fast Fourier Transform (FFT), cepstrum (cepstruam) transform, wavelet transform, cross-correlation method, convolution transform, and the like may be used. The above-described frequency domain transformation method is exemplary and is not limited to the listed transformation methods, and various transformation or analysis methods for analyzing a sound signal in the time domain in the frequency domain may be used.

As an example of the frequency domain data according to embodiments of the present disclosure, spectrogram (spectrogram) data obtained by STFT transformation may be included.

As another example of the frequency domain data according to embodiments of the present disclosure, data obtained by applying a cross-correlation method may be included. In this case, the cross-correlation synthesis of the input data may correspond to a step of inputting the data to the convolution layer, and thus CNN-based learning and classification may be performed using the step.

On the other hand, the frequency domain data for learning may be marked with information required for road classification. In this case, the marking information may include road surface type and/or atmospheric information.

According to an embodiment of the present disclosure, to train the road classification artificial neural network, learning the data set may include marking in the frequency domain data a data set of the road type from which each data is obtained.

The road surface types classified by the road surface classification apparatus according to an embodiment of the present disclosure may include asphalt, cement, soil, ice, marble, paint, slurry (mixed with water), snow, water, and the like. The listed categories are exemplary, and in various embodiments of the present disclosure, the number or set of categories classified by case may vary. Alternatively, instead of using such direct labeling methods or group names, each input data may be grouped in a random manner like the first and second types. Such random grouping may be the result of classification using an unsupervised neural network where the training data does not include tags. But is not limited thereto.

Fig. 3 is a flowchart illustrating a method performed by the road surface classification apparatus according to the present disclosure. According to various embodiments, the operations shown in fig. 3 are not limited to the illustrated order, but may be performed in various orders. Further, according to various embodiments, more operations than those shown in fig. 3 may be performed, or at least one operation less than those shown in fig. 3 may be performed. Fig. 4 to 12 may be referred to as diagrams for explaining the operation shown in fig. 3.

Referring to fig. 3, in step 301, a road surface classification apparatus according to embodiments of the present disclosure may transmit or emit a sound signal to a classification target road surface using a transmitter. In step 301, at least one sound signal may be emitted, and the number of emission times or the emission period of the signal may be changed according to user input, preset conditions, or control of the server. When the sound signal is transmitted a plurality of times in one determination period, since a plurality of data for classification or state determination of the road surface can be acquired, the accuracy of the road surface classification can be improved. A detailed embodiment of the period of transmitting the sound signal and the operation of transmitting a plurality of times in one period will be described below with reference to fig. 4.

In step 302, the road classification device may receive a signal reflected from a target road surface using a receiver. Since the reflected signal is a reflected signal of the emitted sound signal, the sound signal and the reflected signal may correspond to each other. When a plurality of sound signals are transmitted, the reflected signal may be received a plurality of times.

In the case of transmitting the sound signal, the road surface classification device according to the embodiments of the present disclosure may acquire the atmospheric information through the atmospheric sensor of the sensing unit 120. In this case, the timing of acquiring the atmospheric information does not necessarily need to match the timing of sound signal emission, which means that there is a correspondence relationship with each other within a predetermined time interval. That is, the road surface classifying device may acquire the atmospheric information corresponding to one sound signal, or acquire the atmospheric information corresponding to a plurality of sound signals. The road surface classifying device according to the embodiments of the present disclosure may process the reflected signal corresponding to the sound signal based on the atmospheric information corresponding to the emitted sound signal.

On the other hand, the time from when the transmitter transmits one sound signal to when the receiver receives the reflected signal may be defined as time of flight (ToF). Since the propagation speed of sound in the atmosphere can be determined under specific weather conditions, the distance between the road surface classification device and the target road surface can be measured based on the ToF and the atmospheric information. In contrast, when the distance between the road surface classification device and the target road surface is known in advance, the ToF can be estimated. Accordingly, the road surface classification device according to the embodiments can recognize the received signal corresponding to the sound signal transmitted from the transmitter. That is, by determining a reception interval of a received signal corresponding to a sound signal transmitted from a transmitter, a signal received in the interval can be determined as a reflected signal of the transmitted sound signal, and a signal received in other intervals can be regarded as a reflected signal of noise or other sound signals. A detailed embodiment of a control method of the road surface classification device using the control method to control noise signals will be described below with reference to fig. 5.

In step 303, the road surface classification device 100 may preprocess the received reflected signal by the controller to obtain data for input to the road surface classification artificial neural network according to the present disclosure. In the present disclosure, preprocessing of the signal refers to a complete process of acquiring data input to the artificial neural network based on the received reflected signal, and the preprocessing operation in step 303 may include an operation of sampling the analog signal into the digital signal, an operation of correcting an attenuation amount of the sampled signal or ToF correction, a frequency domain transform operation, and the like. A preprocessing process for acquiring input data of the road surface classification artificial neural network according to embodiments of the present disclosure will be described in detail with reference to fig. 6 to 8.

In step 304, the input data acquired through the preprocessing process may be input to the road surface classification artificial neural network. In another aspect, a road surface classification artificial neural network according to embodiments of the present disclosure may be trained as a learning data set comprising a plurality of data acquired for various road surfaces that classify the road surface. The trained pavement classification artificial neural network may output results based on the input data.

The output result according to an embodiment of the present disclosure may include information related to a probability value of each classification category of the road surface. When the artificial neural network model is trained to divide the road surface into a plurality of types, the probability that the target road surface corresponds to each of the plurality of road surface types may be expressed by a numerical value and output. In this case, one or more categories may be output in order of probability of outputting the categories of the road surface.

The output result according to another embodiment of the present disclosure may be output by determining a specific category from a plurality of categories. In this case, the probability value of the corresponding class may be equal to or greater than the threshold value, or the difference between the probability value and the second class may be equal to or greater than the threshold value.

On the other hand, the output result of the road surface classification artificial neural network according to the embodiments of the present disclosure is information related to the material or state of the road surface, and may be output to the user in a necessary form according to the design of the artificial neural network, without being limited to the above example.

In step 305, the road surface classification device may perform various operations according to the output result. By changing or adding the control operation based on the road surface classification result, the accuracy of the result or the efficiency of road surface management can be improved.

According to the embodiments of the present disclosure, when the output result is different from the previous output result, a process of comparing the output result with the previous output result may be performed before controlling the road surface by the output result, so as to distinguish whether the change in the road surface condition is due to the change in the weather condition or the output error. Specific embodiments thereof will be described in detail with reference to fig. 10 and 11. In this case, the road surface classification device may change the transmission frequency or the number of times of the sound signal based on the output result.

According to embodiments of the present disclosure, when the output result is related to a specific category (e.g., snow, ice, or mud), a command or signal for controlling the road surface may be generated and transmitted through the road surface management device. An embodiment of managing the road surface according to the output result will be described in detail with reference to fig. 12.

Referring to fig. 4, the sound signal may be transmitted a plurality of times in one transmission period. In the present disclosure, a set of sound signals transmitted during one transmission period for determining the road surface state is referred to as burst (burst).

The number of sound signals included in one burst may be changed according to a user setting or a predetermined rule. Further, the interval between sound signals included in one burst may be changed according to a user setting or a predetermined rule. The interval between sound signals included in the bursts may be constant or non-constant. Further, the intensities of the sound signals included in one burst may be the same as or different from each other.

In the present disclosure, the number, interval, intensity, and duration of sound signals included in one burst are referred to as burst configuration. In the present disclosure, different bursts may have the same or different burst configurations. The burst configuration of each burst may be changed according to user settings or predetermined rules.

According to an embodiment of the present disclosure, the number of sound signals included in the burst may be one.

According to another embodiment of the present disclosure, the number of sound signals included in the burst may be a plurality.

In the present disclosure, the transmission period refers to a transmission interval of bursts of the road surface classification device classifying the state or material of the target road surface. When a burst consists of one signal, i.e. if only a single signal is transmitted, the transmission period may refer to the time interval between regularly transmitted adjacent sound signals. Referring to fig. 4, the transmission period may correspond to a time interval between the first signal 1a included in the burst 1 and the first signal 2a included in the next burst 2. The transmission period may be changed according to user settings or predetermined rules.

According to embodiments of the present disclosure, the number and/or the transmission period of sound signals included in bursts may be changed according to the result of road classification or weather conditions. For example, in certain weather conditions, such as when snow or air temperature is below freezing, the number of emitted sound signals may be increased or the emission period may be shortened to improve the accuracy of road classification. Specific embodiments thereof will be described in detail with reference to fig. 10 and 11.

On the other hand, according to embodiments of the present disclosure, the emission period may be changed according to the installation position of the road surface classification device or the object. This is to distinguish between reflected received signals from the road surface and crosstalk signals caused by the transmitted signals from the transmitters. The emission period of the road surface classification device installed in the road infrastructure may be longer than that of the road surface classification device installed in the vehicle. Therefore, the determination period of the road surface classification device installed in the road infrastructure may be longer than the determination period of the road surface classification device installed in the vehicle.

According to embodiments of the present disclosure, a road surface classification device may measure or determine the ToF of a target road surface or object. For example, the road surface classification device may transmit one or more sound signals and determine the ToF of the target road surface or object based on the received signals. Alternatively, the ToF may be determined based on the distance between the road surface classification device and the target road surface.

The road surface classification device according to embodiments of the present disclosure may determine an appropriate transmission period and burst configuration based on the determined ToF, and transmit using the determined transmission period and burst configuration. The emission period according to an embodiment of the present disclosure may be set longer than the ToF of the road surface. The duration of the burst according to an embodiment of the present disclosure may be set to be shorter than the transmission period.

In the road surface classification result of the target road surface by the road surface classification device according to the embodiments of the present disclosure, one result corresponding to one burst may be output. Alternatively, the road surface classification device may display classification results for all sound signals included in one burst. When one result is output, the result may be output based on a plurality of classification results for each of a plurality of sound signals included in the burst.

Referring to fig. 4, the first result (result 1) is a road classification result obtained by the first burst (burst 1) based on the reflected signal of the road surface. In this case, the first result may be a result obtained based on the road surface classification result of each of the signals 1a, 1b, 1c, and 1d included in the first burst. For example, the most frequent value in the results of 1a, 1b, 1c, and 1d may be output as the result. Alternatively, the road surface classification result of the first burst may be output based on an average value obtained by adding the results of 1a, 1b, 1c, and 1 d.

In the present disclosure, the time interval between road classification results of adjacent bursts (i.e., the time interval between the first result and the second result) may be referred to as a determination period of road classification. The determined period may coincide with the transmission period. However, in the case of the determination processing, since the output time may be irregular according to the operation of processing the signal, the determination period may not be constant or may not coincide with the transmission period.

The road surface classification device according to embodiments of the present disclosure may change the emission period to change the determination period. Alternatively, the determination period may be changed according to a user setting or a predetermined rule. A specific embodiment of the change determination period will be described in detail with reference to fig. 10 and 11.

Fig. 5 is a diagram showing a transmission period of a sound signal and a reception period of a reflected signal according to an embodiment of the present disclosure.

Referring to fig. 5, the road surface classification device may transmit burst or sound signals in a transmission period. Although the case of transmitting one signal is described for convenience of explanation, the present disclosure is not limited to this description. In other words, it is understood that transmitting an acoustic signal by a road surface classification device in the present disclosure includes transmitting a burst consisting of a plurality of signals in a period and transmitting one signal in a single manner.

As described above, the road surface classification device according to the embodiments of the present disclosure may determine the ToF of the road surface of the transmitted sound signal, and thus may determine the corresponding reception interval in advance for one transmission interval.

According to embodiments of the present disclosure, when a receiver senses a signal before a reception interval, a road classification device may treat it as a noise signal or a crosstalk signal, and may control a transmitter of the road classification device to reduce the signal.

Specifically, referring to fig. 5, when the intensity of the first signal received before the reception interval is greater than a first threshold value, or when the difference between the intensity of the second signal received within the reception interval and the intensity of the first signal received before the reception interval is less than a second threshold value, the power supplied to the transmitter may be changed to control this. The first threshold and/or the second threshold may be predetermined or set by a user input or an external device.

For example, when the intensity of the first signal is greater than the first threshold, it may be determined that the influence of crosstalk is large, and control may be performed to reduce vibration of the transmitter. Alternatively, when the intensity of the second signal is smaller than that of the first signal, it may be determined that noise caused by the external environment is larger than the received signal, and control may be performed to increase the vibration of the transmitter. According to an embodiment of the present disclosure, vibration of the transmitter may be controlled by adjusting a power value supplied to the transmitter.

Referring to fig. 6, the road surface classifying device 100 according to embodiments of the present disclosure may be installed in a moving object 610 or a road infrastructure 620. In this case, the road surface classification device 100a mounted on the moving object 610 (e.g., a vehicle) and the road surface classification device 100b mounted on the road infrastructure 620 have different heights for the road surface, and thus the ToF of the emitted sound signals is different.

As sound propagates in space, the amplitude decreases with increasing distance from the sound source and if the sound propagates in air, it is attenuated by the medium. Therefore, even in the same state, the characteristics of the reflected signal of the road surface having different ToF may be different.

On the other hand, the road surface classification apparatus of the present disclosure uses an artificial neural network to classify the road surface based on the reflected signal of the road surface, and thus a large number of data sets are required to train the artificial neural network.

Referring to fig. 7, a learning data set of an artificial neural network for learning a road surface classification according to embodiments of the present disclosure may be acquired for various road surfaces of each road surface classification (category) using a transceiver included in a mobile measurement device 700. In general, it is important to collect a large amount of data in various terrains and environments to improve the classification performance of artificial neural networks, and for this reason, it is important to collect data using devices that are easy to move.

The movement measurement apparatus 700 of the present disclosure refers to a sensor device mounted on an apparatus (e.g., a bicycle, an automobile, or a scooter) that moves on a road, and may include an apparatus that can be moved by a person or a mechanical device.

On the other hand, the ToF of the learning data collected by the movement measuring device 700 may be similar to the ToF of the road surface classification apparatus 100a mounted in the moving object 610 such as the vehicle of fig. 6. Alternatively, the position of the mobile measuring device with respect to the ground may be set in consideration of the position of the road surface classification device mounted in the mobile object with respect to the ground. In this case, the road surface classification artificial neural network trained using the learning data set acquired by the mobile measurement device according to the embodiments of the present disclosure may be directly used for the road surface classification apparatus 100a installed in the moving object without any additional correction of the reflected signal.

However, as shown in fig. 6, when the road classification device is installed at a different height from a moving object such as a road infrastructure (100 b) and a reflected signal acquired by the road classification device is directly input to the road classification artificial neural network, classification accuracy of a target road surface may be lowered.

Fig. 8 is a flowchart illustrating a process of preprocessing a received reflected signal by a road surface classification device according to various embodiments. The preprocessing process of fig. 8 is an example for expressing the technical idea of the present disclosure, and according to various embodiments, more operations may be performed or at least one operation may be performed less than the operations shown in fig. 8.

In operation 801, a road surface classification apparatus according to embodiments of the present disclosure may acquire first data based on a received reflected signal.

As described above, the reflected signal received through the receiver may be an analog signal, and thus, the road surface classification apparatus of the present disclosure may convert the reflected signal into a digital signal through the ADC circuit included in the controller 130. Alternatively, a transceiver included in a road surface classification apparatus according to an embodiment of the present disclosure may process a reflected signal reflected by a road surface in the form of a digital signal to acquire first data.

In operation 802, the road surface classification apparatus according to embodiments of the present disclosure may acquire second data by applying an atmospheric correction to the data converted into a digital signal.

The sound propagates in air and is attenuated by the influence of the medium, the attenuation being determined by the propagation distance and the attenuation coefficient. On the other hand, since the attenuation coefficient is a value determined based on the temperature, humidity, air pressure, and frequency of the sound signal, the road surface classification apparatus according to the embodiments of the present disclosure may calculate the attenuation amount of the sound signal based thereon.

The road surface classification device according to the embodiments may generate the second data by correcting the attenuation of the received reflected signal based on the atmospheric information such as temperature, humidity, air pressure, etc., which is acquired by the atmospheric sensor.

On the other hand, the sound propagation distance required for the atmospheric correction may be input in advance by the user, or may be acquired based on the ToF. That is, the distance information of the road surface may be input in advance according to the installation position of the road surface classification device, or as described above, the distance information of the road surface may be acquired based on the ToF information and the atmosphere information acquired by the road surface classification device.

In step 803, the road surface classification device according to the embodiments of the present disclosure may acquire the third data by applying the distance correction to the second data correcting the attenuation amount in the atmosphere.

As described above, referring to fig. 6 and 7, the sound signal, which is the basis of the learning data of the road surface classification artificial neural network according to the embodiments of the present disclosure, may be a signal obtained by reflection at a distance d1 from the road surface. Therefore, in order to improve the classification performance of the road surface classification device 100b installed at the height d2 different from d1, the sound signal acquired at d2 may be corrected to the sound signal acquired at d 1.

On the other hand, steps 802 and 803 may be performed in one process. With respect to the digital signal obtained in step 801, sound data in which the amount of atmospheric attenuation and the distance to the road surface are corrected based on the atmospheric information and the distance information can be obtained.

Further, according to the installation position of the road surface classification device according to the embodiments of the present disclosure, the correction process of steps 802 and/or 803 may be omitted.

In step 804, the road surface classification apparatus according to embodiments of the present disclosure may acquire frequency domain data by performing transformation to analyze corrected sound data in the frequency domain. The frequency domain transform method according to embodiments of the present disclosure is as described above. The obtained frequency domain data is input data for the road surface classification artificial neural network according to the embodiments of the present disclosure, and when the frequency domain data is input to the road surface classification artificial neural network, the road surface classification artificial neural network may output a road surface classification result of the target road surface.

The road surface classification device according to the embodiments of the present disclosure may output the result based on additional information other than the sound signal. Other information that may be additionally acquired in addition to the sound signal may include image information acquired by a vision sensor (camera), road surface temperature information acquired by an infrared sensor, and environmental information acquired by a communication unit.

According to an embodiment of the present disclosure, the road surface classification device may combine more than two different determination criteria.

Since the region identifiable by sound may correspond to a partial region of the road surface, image information identifiable in a wider region may be used to assist the road surface classification result. For example, only when the result of the identification by the image information matches with the output value of the road surface classification artificial neural network, it can be determined as effective road surface information. On the other hand, the road surface classification apparatus according to embodiments of the present disclosure may further include a separate image-based road surface classification artificial neural network for obtaining the road surface classification result of the image information.

Further, according to various embodiments, the road surface classification device may verify the resulting value of a particular road surface state by applying a particular temperature condition. For example, when the road surface temperature is higher than 0 degrees celsius, ice cannot be physically formed at atmospheric pressure, and thus when the road surface classification result is classified as freezing under the corresponding temperature condition, it may be determined to correspond to an error. Accordingly, when it is confirmed that the road surface or the atmospheric temperature acquired through the infrared sensor or the atmospheric sensor of the sensing unit is higher than a specific temperature, if the road surface state indicated by the road surface classification result is related to ice, an additional operation may be performed instead of outputting the corresponding result. Alternatively, when the road surface temperature is higher or lower than a specific temperature, the result may be further used for image information to output the result.

Further, according to embodiments of the present disclosure, the road surface classification device may output the road surface classification result in consideration of weather environment information. For example, when weather environment information related to weather is received, for example, when snow or rain is put down, the rank of the road surface classification result of the category having a higher likelihood of being classified in the corresponding weather may be adjusted.

On the other hand, since the above-described image information and temperature information are useful information for classifying the road surface state, the road surface classification artificial neural network can enhance learning performance and classification performance by receiving the relevant additional data together instead of learning only with data based on sound signals.

Referring to fig. 9, a road surface classification artificial neural network according to embodiments of the present disclosure may include a multi-modal artificial neural network. In addition to input data related to the sound signal, the multi-modal artificial neural network may function as one classifier by inputting at least one of image information, atmospheric information, or road surface temperature information together through a classifier based on different information. By such correspondence learning, a plurality of pieces of information related to one road surface state are input together, and a more accurate road surface classification result can be obtained. That is, the road surface classification device according to the embodiments of the present disclosure may output the road surface result by combining a plurality of pieces of information. On the other hand, the input data shown in fig. 9 is exemplary, and only a part of the image information, the atmospheric information, and/or the road surface temperature information may be utilized, or additional information may also be utilized.

Fig. 10 is an operation flowchart showing a change of a control operation of the road surface classification apparatus according to an embodiment of the present disclosure based on a predetermined control change trigger.

In the present disclosure, the control change trigger refers to a case or condition for changing the operation of the road surface classification device according to the embodiments of the present disclosure, and may be preset by a user or set by a command from an external device.

On the other hand, changing the control operation of the road surface classification device by the control change trigger means that the scheme set before the control change trigger occurs in the road surface classification device is changed, such as burst configuration, emission period, determination period, road surface classification result output scheme, and the like.

Control change triggers may include, but are not limited to, a change in road classification results (categories) or an output of a particular category, weather conditions, time conditions, geographic conditions, and the like.

As an example of the control change trigger, when the road classification result is changed, the operation of the road classification device may be changed.

Referring to fig. 11, the road surface classification result of the road surface classification device according to an embodiment of the present disclosure may be changed to a first class (R1) at a first time (t 1) and to a second class (R2) at a second time (t 2). The second category may be a different category than the first category.

According to an embodiment of the present disclosure, the second category may represent a road surface state related to ice, and the first category may be a classification result related to other road surface states.

The road surface classification device according to the embodiments of the present disclosure may require a change control operation when the road surface state changes.

For example, when the road classification result is different from the previous classification result, the road classification device may change the transmission period or change the burst configuration to check whether the road classification result is erroneous. That is, the number of determinations may be increased by shortening the transmission period, or the number of sound signals included in the burst may be increased. Alternatively, instead of directly changing the transmission period or burst configuration, it may be determined whether to change the transmission period or burst configuration based on a subsequent determination.

Referring to fig. 11, since the result of the second time is different from the result of the first time (r1+noter2), the road surface classification apparatus according to an embodiment of the present disclosure can control the emission period to make a shorter change, and can obtain more results in a shorter time interval (t 3 to t 6). On the other hand, the number of determinations of the second class is larger than the number of determinations of the first class in the corresponding time interval (t 3 to t 6), so the road surface classification device can determine that the determination of the second class is accurate, change the emission period to the original state, and output the result at a seventh time (t 7) which is a time based on the changed emission period.

According to embodiments of the present disclosure, as an example of changing a control operation based on a subsequent determination when a change is previously determined, when a first category (R1) is determined at a first time (t 1) and a second category (R2) different from the first category is determined at a second time (t 2), it may be determined whether to change the determination accuracy or the transmission period of the second category not immediately but based on a result of a third category which is a third time (t 3) of the next determination time. That is, when the result of the third category is determined as r2, the category that is changed at the second time may be determined to be accurate and the transmission period may not be changed, whereas when the result of the third category is not determined as r2, the determination of r2 may be determined to be erroneous and the transmission period may be changed.

On the other hand, according to embodiments of the present disclosure, the determination result of each time may be a result corresponding to each burst.

The road surface classification device according to the embodiments of the present disclosure may also perform operations related to managing the road surface based on the results of a plurality of determinations in the changed control operation. For example, after determining the accuracy of a particular category of change, the relevant category may be deemed correct and the pavement management operations associated with the changed category may be performed. Details relating to the road surface management operation will be described with reference to fig. 12.

As another example of the control change trigger, when the weather condition or the time condition is changed, the operation of the road surface classification device may be changed.

As an example of weather conditions, when the temperature is lower than zero degrees celsius or when the temperature is predicted to be lower than zero degrees celsius, the road surface classification apparatus according to the embodiments of the present disclosure may change the control operation by shortening the emission period or increasing the number of emissions within a burst to quickly determine whether or not black ice is present. As another example of the weather conditions, there may be a weather environment such as strong wind, heavy rain, heavy snow, or the like, and the information related to the weather environment may be obtained through an atmospheric sensor included in the road surface classification device, or may be obtained from an external device through a communication unit.

As an example of a time condition, since black ice may occur more easily during night than during day, the transmission period or burst configuration may be changed at a specific time in consideration of this. As another example of the time condition, since the change in the road surface state in summer may be less than in winter, the control operation may be changed by extending the transmission period or reducing the number of transmissions in a burst to reduce the power consumption.

As another example of a control change trigger, the operation of the road surface classification device may be changed when the geographic condition changes. When the road surface classification device is installed in a road infrastructure, the geographical conditions may not change, but the transmission period or burst configuration may be different for each area condition. When the road surface classification device is installed in a moving object such as a vehicle, the transmission period or burst configuration may be different when entering a specific area. For example, when the road surface classification device receives the related information when entering a black ice easy-to-get road section, the control operation may be changed as described above.

On the other hand, the control change trigger related to the weather condition, the time condition, or the geographical condition described above is exemplary, and is not limited thereto, and may be set by a user input or a signal received from an external electronic device such as a server device.

Referring to fig. 12, in step 1201, the road surface classification apparatus may obtain information related to control of the road surface. According to embodiments of the present disclosure, the information related to the control of the road surface may include the result obtained in step 304 of fig. 3 or the final result obtained through the change control operation of fig. 10. Further, the information related to the control of the road surface may include weather information and/or road surface temperature information obtained by the road surface classification device.

In step 1202, the road surface classification device according to the embodiments may determine whether the operation condition of the road surface control is satisfied based on the obtained information.

For example, when the road surface classification result obtained in step 1201 is related to black ice, that is, if the class related to ice is obtained, the road surface classification device may determine that the condition for operating the road surface management device mounted on the road surface is satisfied to eliminate or prevent the frozen state of the road surface.

Alternatively, the operation condition of the road surface control may be determined by combining the obtained road surface classification result and weather information. For example, if the weather information satisfies a specific condition, it may be determined that there is a high risk of icing, and a condition that activates a road surface management device installed on the road to clear or prevent an icing condition on the road has been satisfied.

Examples of the weather information for operating the road surface management device may include the following conditions.

(1) If snow, rain, snow or frost is predicted or snow, rain, snow or frost is predicted,

(2) If the road surface temperature is below zero,

(3) If the time is dawn,

(4) If the temperature is rapidly lowered down to a value,

(5) If the wind is blown out in a large amount,

When at least one of the above weather conditions is satisfied, if the road surface classification result obtained is a specific category (e.g., water, mud, ice), it may be determined that road surface control is required.

The pavement management apparatus according to embodiments of the present disclosure may include a saline spray apparatus or a heating wire apparatus, but is not limited thereto.

In step 1203, the road surface classification apparatus according to the embodiments of the present disclosure may generate a road surface control signal based on the obtained information and the determination. The road surface control signal may include signals or commands required to control a road surface management device mounted on the road surface.

The road surface classifying device according to the embodiments of the present disclosure may be interlocked with a road surface managing device installed on a road. When the road surface classification device is directly interlocked with the road surface management device, the road surface classification device may generate a command signal for controlling the road surface management device and transmit the command signal to the road surface management device. Alternatively, when the road surface classification device is indirectly interlocked with the road surface management device through the external server, the road surface classification device may generate a signal indicating control of the road surface management device and transmit the signal to the external server.

On the other hand, when the road surface control signal is received by the road surface management device, an operation of controlling the road surface may be performed based on the road surface control signal. For example, the pavement management device may spray brine or operate the heater wire device based on the pavement control signal.

As another embodiment of the pavement management of the present disclosure, the pavement classification device may determine the risk of pavement damage.

Asphalt may be damaged by repeated passes of vehicles exceeding a certain weight. In particular, when water permeated between asphalt is frozen, the volume may be enlarged, and when a large vehicle such as a truck passes, the road surface may be damaged.

According to embodiments of the road surface classification device installed on the road infrastructure, the ToF of the road surface can be periodically sensed, and thus traffic information such as traffic vehicle information and traffic volume can be measured.

Referring to fig. 13, the road surface classification device may collect traffic information on the road surface based on the measurement of ToF. Traffic information collected by the road surface classification device according to embodiments of the present disclosure may include information related to the degree of road surface damage or traffic volume.

As described above, since the road surface classification device can obtain information about the installation height of the road surface classification device, the ToF corresponding to the installation height of the road surface classification device can be determined as the reference ToF. That is, the ToF of the reflected signal reflected from the road surface may be referred to as a reference ToF.

Accordingly, when the ToF obtained by the road surface classification device is identified as corresponding to the reference ToF, the road surface classification device according to an embodiment of the present disclosure may determine that there is no vehicle on the road surface. Further, when the obtained ToF is recognized as being shorter than the reference ToF, it can be determined that a vehicle is present on the road surface. Further, information about the size (height) of an object on the road surface estimated based on the obtained ToF may be obtained.

Since a shorter ToF value may mean that there is an object that is higher from the road surface, the road surface classification device may determine that a large vehicle has passed by a signal with a short ToF. The criteria for determining the large vehicle may be preset by a user input or a signal of an external device.

The road surface classification device according to the embodiments of the present disclosure may estimate the amount of traffic passing through the road surface during a predetermined time interval based on the ToF value obtained during the time interval. Further, the traffic amount information obtained by the road surface classification device of the present disclosure may also include information related to the size of the passing vehicle.

Referring to fig. 13, toF1 of (a) through which a large vehicle is passing is measured to be smaller than ToF2 of (b) through which a small vehicle is passing.

The road surface classification device according to the embodiments of the present disclosure may obtain road surface state information and/or weather information, and thus may determine the risk of road surface damage in combination with the obtained information and traffic information, and notify the outside. The road surface state information may include road surface classification results and/or road surface temperature information.

For example, information on the number of passes of the large vehicle may be measured during a period in which the road surface classification result is determined to be a category related to ice, and information related thereto may be provided to a user or an external device. The external device may include a server device that manages an organization of the link. Alternatively, information about the traffic volume of the vehicle may be obtained during a period in which the road surface temperature is measured to be below a certain temperature, and may be transmitted to an external device. In this case, the degree of damage of the road surface may be estimated based on the result of the classification of the specific road surface or the traffic volume of the large vehicle under specific weather conditions. Alternatively, by providing the obtained information on the traffic volume of the large vehicle to the external device, the risk of damaging the road surface may be managed.

The road surface classification device according to the embodiments of the present disclosure may determine whether to use road surface classification or to collect traffic information based on the ToF of the acquired reception signals. That is, when the ToF of the acquired reception signal is within the error range from the reference ToF, it may be determined as a reflected signal of the road surface and used for road surface classification, and when the ToF of the acquired reception signal is shorter than the reference ToF, it may be determined that it is acquired from the vehicle, and traffic information may be acquired based thereon.

In order to collect traffic information, the road surface classification device may shorten the transmission period of the sound signal as compared with the transmission period for road surface classification. That is, the transmission period of the sound signal may be set differently according to the needs of the user, and the acquired signal may be processed differently according to purposes.

In another aspect, the road surface classification device of the present disclosure may include a road surface type estimation device. It is apparent that the operations performed by the road surface type estimating apparatus according to the embodiments of the present disclosure may be performed by the road surface classifying apparatus according to the embodiments of the present disclosure.

As shown in fig. 14, a road surface type estimation device using sound according to an embodiment of the present disclosure may include a sound transceiver 1410, a signal converter 1420, an artificial neural network 1430, and a controller (MCU) 1440. On the other hand, the road surface type estimation device may further include an atmospheric attenuation correction unit (not shown) and an atmospheric information measurement unit (not shown).

The acoustic transceiver 1410 may transmit an acoustic signal to a corresponding road surface to learn the type and then receive a reflected signal.

The acoustic transceiver 1410 may include an acoustic transmitter 1411 that outputs a transmission signal and an acoustic receiver 1412 that receives a reflection signal returned by the transmission signal being reflected by any surface under the control of a controller 1440.

The signal converter 1420 may perform frequency conversion on a predetermined region in a time domain of the received signal to acquire a frequency domain signal (e.g., a spectrogram).

The signal converter 1420 may include a short-time fourier transform (STFT) converter, a Fast Fourier Transform (FFT) converter, a cepstral (capstruam) or a wavelet transform. In this case, the frequency domain signal (spectrogram) may be 2D or 3D.

The artificial neural network 1430 may take the frequency domain signal (spectrogram) as an input signal, extract features of the input signal based on a trained road classification model, and estimate the type of road.

On the other hand, the signal converter 1420 may include an analog-to-digital converter (ADC). The ADC may convert an analog signal of the received signal into a digital signal.

An atmospheric attenuation correction unit (not shown) may calculate and correct the atmospheric attenuation of the digital signal.

The artificial neural network 1430 may perform convolution on the input signal based on the trained road classification model and estimate the type of road surface from the classification.

On the other hand, the artificial neural network 1430 may use at least one or more of decision trees, linear discriminant analysis, logistic regression classifiers, naive bayes classifiers, support vector machines, nearest neighbor classifiers, and ensemble classifiers for classification and learning.

Decision trees may include fine, medium, coarse, all and optimizable trees, discriminant analysis includes linear discriminants, quadratic discriminants, all discriminants and optimizable discriminants, na iotave bayes may include gaussian na iotave bayes, kernel na iotave bayes, all na iotave bayes and optimizable na iotave bayes, support Vector Machines (SVMs) include linear SVMs, quadratic SVMs, cubic SVMs, fine gaussian SVMs, medium gaussian SVMs, coarse gaussian SVMs, all SVMs and optimizable SVMs, nearest neighbor classifiers may include fine KNN, medium KNN, coarse KNN, cosine KNN, cubic KNN, weighted KNN, all KNN and optimizable KNN, and integrated classifiers may include lifting trees, bagging trees, subspace discriminants, subspace KNN, RUSBoosted trees, all integrations and optimizable integrations.

A controller (MCU) 1440 may control the operation of the sound transceiver 1410, the signal converter 1420, and the artificial neural network 1430.

The signal converter 1420 and the artificial neural network 1430 are represented as components by software implemented in a program.

On the other hand, although not shown in the drawings, the road surface type estimation device using sound signals according to the embodiments of the present disclosure includes a Memory (Memory) in which a trained road surface classification model and software implemented in a program are stored. A Memory (Memory) may be included in the controller (MCU).

On the other hand, the road surface type estimation device using sound according to the present disclosure may further include an atmospheric sensor (not shown) capable of measuring temperature, humidity, and atmospheric pressure in the air.

Atmospheric information including temperature, humidity, and atmospheric pressure may be used in the atmospheric attenuation correction unit or transmitted to an input of the artificial neural network 1430 according to the control of the controller 1440.

Fig. 15 is a diagram for explaining a transmission signal and a reception signal in a road surface type estimation device using sound signals according to an embodiment of the present disclosure.

As shown in fig. 15, a controller (MCU) 1440 may transmit a trigger signal having a predetermined amplitude (v: trigger voltage) and having a predetermined transmission period (p: transmission period) to the acoustic transceiver 1410.

Then, the sound emitter 1411 of the sound transceiver 1410 may output the sound signal 1501 having a specific frequency (e.g., 40 kHz) to the corresponding road surface to learn the type.

Thereafter, the sound receiver 1412 of the sound transceiver 1410 may receive the reflected signal back to the road surface.

Here, the signal 1502 received in the same time line as the sound signal 1501 may be a crosstalk signal of the sound signal transmitted by the sound transceiver 1410. Further, the controller 1440 may determine the signal 1503 as a received signal during a predetermined time from a point where the amplitude received after the transmission delay is maximum.

For example, when the time at which the amplitude of the signal received after the crosstalk signal is maximum is t_0, the total (a+b) ms from t_0-a ms to t_0+b ms can be observed, and in the reception signal 203 of fig. 15, a is 0.2 and b is 5. A and b are variable values that can be adjusted depending on the circumstances or conditions.

In the example of fig. 15, 10ms is used as 1 transmission period, 10ms is a time until the transmitted sound signal sufficiently disappears, and the sampling frequency of the sound transceiver 1410 is 1mhz, which is 25 times the frequency of the 40kHz sound.

On the other hand, as described above, a plurality of received signals may be sensed according to a transmission period to sense the state of the road surface, or a single reflected signal received after transmitting a sound once may be processed to sense the state of the road surface.

Fig. 16 is a diagram for exemplarily explaining a signal converter in a road surface type estimation apparatus using sound signals according to an embodiment of the present disclosure.

In fig. 16, an example of using an STFT converter as the signal converter 1420 will be described.

As shown in fig. 16, the STFT converter may obtain a 2D spectrogram 1602 by performing short-time fourier transform on signals 1503 and 1601, which are received during a predetermined time period after a transmission delay, other than the crosstalk signal 1502 of the sound signal transmitted by the sound transceiver 1410, of the reflected signals received in fig. 15.

The signal 1503 of one cycle may be fourier transformed, or the received signal of a plurality of cycles may be fourier transformed.

In the present disclosure, acoustic impedance and surface roughness information may be used to distinguish materials. The acoustic impedance is not constant and the value may vary for each frequency of sound vibrations. Thus, frequency domain analysis may be useful. Fourier transforms, which are one of various methods for converting a time domain received signal into a frequency domain signal, may be used. Further, a short time fourier transform may be used to confirm the FFT for each (sample time).

Furthermore, frequency analysis may be performed using wavelets, and in one embodiment of the present disclosure, the use of STFT to reduce the amount of computation and ensure adequate data is illustratively described.

The Short Time Fourier Transform (STFT) is a method designed to take into account time variations that have not been resolved in existing fourier transforms. STFT is to divide a long signal varying with time in units of a short time and then apply fourier transform.

Since the STFT separates signals according to window lengths, the length of the signal for fourier transform is reduced, and thus the resolution of frequency may be deteriorated. On the other hand, when the window length is increased to increase the resolution of the frequency, the temporal resolution may be deteriorated. Wavelet Transforms (WTs) may be used to overcome resolution limitations due to the balanced relationship between frequency and time.

If the window length is determined in the STFT, the STFT is performed a plurality of times while changing the window length. Furthermore, if a sinusoidal curve extending to infinity in time is used as a basis function in the STFT, the wavelet has various functions with a finite period. The wavelet functions may include Morlet, daubechies, coiflets, biorthogonal, mexican Hat, symlets, etc.

As shown in fig. 17, the artificial neural network may include a Deep Neural Network (DNN) of a multi-layer perceptron algorithm that includes an input layer 1701, a plurality of hidden layers 1702, and an output layer 1703. Further, the artificial neural network according to embodiments of the present disclosure may include a Deep Convolutional Neural Network (DCNN) further having a multi-layer perceptron algorithm of a convolutional execution unit (not shown).

The input layer 1701 may planarize the data for the spectrogram 1702 and receive the 1D flattened data (FLATTENED DATA).

Data input to the input layer 1701 may be characterized and categorized by a plurality of hidden layers 1702.

The output layer 1703 may output probability values for each trained road surface.

The artificial neural network can determine and output the kind of road surface having the highest probability among the probability values output from the output layer 1703 by using the softmax 1704.

On the other hand, the artificial neural network may receive atmospheric information (temperature, humidity, and pressure information) and may use it as an input to the input layer 1701.

In addition, the artificial neural network may also fourier transform the sound signal 1501 emitted to the road surface, and may use it as an input to the input layer 1701.

On the other hand, when the artificial neural network is DCNN, the convolution performing unit may perform convolution operations on the received digital input signal a plurality of times, perform batch normalization, reLU function, and MaxpPooling function on each convolution operation, and output the flattened data in the last convolution operation to the transmission layer.

In this case, the transmission layer is an input layer 1701 of CNN, and flattened output data of the convolution execution unit can be received in one dimension (1D), and the subsequent operations are the same as described above.

The convolution performing unit may perform a plurality of (e.g., 5) 1D convolution operations on the input signal, perform a batch normalization, a ReLU function, and MaxpPooling functions on each convolution operation, and the output of the last convolution operation may be the flattening data.

Referring to fig. 18, for example, the input signal 1801 may be about 7000 received signals during 7ms, the first convolution execution result 1802 may be a result of performing 1D convolutions (64, 16), BN, reLU, and MP (8) on the input signal 1801, the second convolution execution result 1803 may be a result of performing 1D convolutions (32, 32), BN, reLU, and MP (8) on the first convolution execution result 1802, the third convolution execution result 1804 may be a result of performing 1D convolutions (16, 64), BN, reLU, and MP (8) on the second convolution execution result 403, the fourth convolution execution result 1805 may be a result of performing 1D convolutions (8, 128), BN, and ReLU on the third convolution execution result 1804, and the fifth convolution execution result 1806 may be a result of performing 1D convolutions (4, 2568), BN, and ReLU on the fourth convolution execution result 1805.

Fig. 19 is a diagram showing codes of a convolution execution unit of the road surface type estimation device using sound according to the present disclosure.

Fig. 19 is code implementing a portion of the convolution execution unit shown in fig. 18 in software.

These codes may include a plurality of BatchNorm functions and MaxPool functions.

As one example of an embodiment of the present disclosure, one-dimensional convolution is first performed, then one-dimensional batch normalization is performed, then max pooling is performed, four groups are repeated, and finally the probability of each road surface is output by outputting as many values as the number of road surfaces to be classified by the full connected layer.

On the other hand, in the present disclosure, a method of performing a one-dimensional (1 d) convolution operation is described as an example, but the convolution operation may be 2d and 3d and 1d.

First, in order to perform the road surface type estimation method using domain transformation of sound according to the present disclosure, first, a learning step 2001 may be performed in advance to generate a road surface classification model.

In the learning step 2001, for a plurality of road surface types, after transmitting the sound signal, the reflected signal is received, the signal is converted into a frequency domain signal (e.g., spectrogram), and the frequency domain signal (spectrogram) is input to the artificial neural network to train the road surface classification model.

Here, in order to convert the frequency domain signal, STFT (short time fourier transform), FFT (fast fourier transform), cepstrum, or wavelet transform may be used. The frequency domain signal may be 2D or 3D.

Thereafter, under the control of the controller, acoustic signals may be transmitted to the corresponding road surface to learn the type, and reflected signals may be received (2002).

Thereafter, under the control of the controller, signal conversion may be performed on a preset region of the received signal to obtain a frequency domain signal (2003).

In the frequency domain signal acquisition step 2003, a frequency domain signal may be obtained by performing domain conversion on a signal (a crosstalk signal excluding a transmitted sound signal) during a predetermined time period received after a transmission delay of each period of the sound signal.

Thereafter, under control of the controller, the frequency domain signal may be an input signal of an artificial neural network, and features of the input signal may be extracted and classified based on the trained road surface classification model to determine a type of road surface (2004).

The artificial neural network may include a Deep Neural Network (DNN) of a multi-layer perceptron algorithm that includes an input layer 1701, a plurality of hidden layers 1702, and an output layer 1703. In this case, the output layer 1703 may output a probability value for each trained road surface type, and the artificial neural network may determine the road surface type with the highest probability by using the softmax 1704.

The artificial neural network may receive atmospheric information (temperature, humidity, and barometric pressure information) and may be used as an input layer.

In addition, the artificial neural network may also convert sound signals transmitted to the road surface into fourier transforms and may serve as an input layer.

First, in order to perform the method of estimating the road surface type using sound according to the present disclosure, the learning step 2101 may be performed in advance to generate a road surface classification model.

In the learning step 2101, sound signals may be transmitted to various types of road surfaces, reflected signals may then be received, and the corresponding signals may be converted into digital signals, and the converted digital signals may be input to an artificial neural network to perform multiple convolution operations to train a road surface classification model.

Thereafter, under the control of the controller, the sound signal may be transmitted to the corresponding road surface to learn the type, and then the reflected signal may be received (2102).

Thereafter, the analog signal may be converted into a digital signal for a preset area of the received signal under the control of the controller (2103).

In the signal conversion step 2103, in the case of not including the crosstalk signal of the transmitted sound signal, for each period of the sound signal, the signal during the predetermined time period may be converted into a digital signal based on a point at which the amplitude of the received signal after the transmission delay is maximum.

For example, when the time at which the amplitude of the signal received after the crosstalk signal is maximum is t_0, a total of (a+b) ms from t_0-a ms to t_0+b ms may be observed, and a and b may be variably adjusted according to the environment or conditions.

Thereafter, under the control of the controller, the digital signal may be received to perform a plurality of convolution operations on the artificial neural network (2104).

In the convolution operation step 2104, a plurality of convolution operations may be performed on the digital signal, and batch normalization, reLU functions, and MaxpPooling functions may be performed on each convolution operation, and the output of the last convolution operation is flat data.

On the other hand, in the present disclosure, a method of performing a one-dimensional (1D) convolution operation has been described as an example, but the convolution operation may be 2D and 3D as well as 1D.

Thereafter, under the control of the controller, based on the road surface classification model trained by the artificial neural network, the features of the convolution calculation signal (flattening data) may be extracted and classified to determine the type of road surface (2105).

The artificial neural network may include: a convolution performing unit that receives the digital signal and performs a plurality of convolution operations; and a Deep Convolutional Neural Network (DCNN) of a multi-layer perceptron algorithm, the deep convolutional neural network of the multi-layer perceptron algorithm comprising a transport layer, a plurality of hidden layers, and an output layer. In this case, the output layer may output a probability value for each road surface type trained, and the artificial neural network may determine and output the road surface type with the highest probability using softmax.

The artificial neural network may receive the atmospheric information (temperature, humidity, barometric pressure information) and may use it as an input to the convolution execution unit.

In addition, the artificial neural network may use a sound signal transmitted to the road surface as an input of the convolution performing unit.

Fig. 22 is a flowchart illustrating a method of estimating a road surface type using sound with a corrected air attenuation amount according to an embodiment of the present disclosure.

First, in order to perform the method of estimating the road surface type using sound according to the present disclosure, a learning step (2201) may be performed in advance to generate a road surface classification model.

In the learning step 2201, for a plurality of road surfaces, sound signals are transmitted, reflected signals are received, corresponding signals may be converted into digital signals, the converted digital signals may be subjected to atmospheric attenuation correction and converted into frequency domain signals, and the frequency domain signals are input to a neural network to learn a road surface classification model.

Here, in the learning step 2201, in order to convert the frequency domain signal, STFT (short time fourier transform), FFT (fast fourier transform), cepstrum, or wavelet transform may be used. The frequency domain signal may be 2D or 3D.

Thereafter, based on the control of the controller, the sound signal may be transmitted to the corresponding road surface of which the type is desired to be known, and the reflected signal may be received (2202).

Thereafter, based on the control of the controller, an analog signal of a predetermined area of the received signal may be converted into a digital signal (2203).

In the signal conversion step 2203, in the case of a crosstalk signal that does not include the transmitted sound signal, for each period of the sound signal, a signal during a predetermined time period may be converted into a digital signal based on a point at which the amplitude of the received signal after the transmission delay is maximum.

Thereafter, the air-fade amount of the digital signal can be calculated and corrected by the control of the controller (2204).

In the atmospheric attenuation correction step (2204), the atmospheric attenuation amount may be calculated and corrected using the following equations 1 to 8. The atmospheric attenuation correction step may be performed by an atmospheric attenuation correction unit, which is software implemented by a controller or a program.

First, the saturation pressure (Psat) can be calculated using the following < equation 1 >.

[ Equation 1]

Here, to1 is the triple point K of the atmosphere, and T is the current temperature [ K ].

Absolute humidity (h) can be calculated using the following < equation 2 >.

[ Equation 2]

h＝h_rinP_sat/P_s

Here hrin is the relative humidity [% ], psa is the saturation pressure [ unit ], and Ps is the constant pressure [ atm ].

On the other hand, the scaled relaxation frequency (FrN) of nitrogen, which is 78% of the atmosphere, can be calculated using the following < equation 3 >.

[ Equation 3]

Here, to is the reference temperature [ K ], and T is the current temperature [ K ].

On the other hand, the scaled relaxation frequency (FrO) of oxygen, which is 21% of the atmosphere, can be calculated using the following < equation 4 >.

[ Equation 4]

Here, h is absolute humidity.

On the other hand, the attenuation coefficient (α: attenuation coefficient [ nepers/m ]) can be calculated using the following < equation 5 >.

[ Equation 5]

Here, ps is a constant pressure, F is a frequency of a sound signal (emitted sound signal), T is a current temperature K, to is a reference temperature K, frO is a relaxation frequency of scaled oxygen, and FrN is a relaxation frequency of scaled nitrogen.

On the other hand, the attenuation rate (A, [ unit: dB ]) of the sound signal can be calculated using the following < equation 6 >.

[ Equation 6]

A＝10·log₁₀exp(2α)·d

Here, α is an attenuation coefficient, and d is a distance between the acoustic transceiver 100 and a corresponding road surface of which the type is intended to be known.

D can be calculated using the following < equation 7>, where t (time of flight) from the time the signal is transmitted from the transmitter to the time the signal is reflected by the road surface and detected by the receiver, and the speed of sound (Vair) in the atmosphere are used.

[ Equation 7]

d＝V_air*t/2

Here, t is the time of flight, and Vair is the atmospheric sound velocity [ m/s ].

On the other hand, the atmospheric sound velocity can be calculated using the following < equation 8 >.

[ Equation 8]

Here, ks is the isentropic volumetric expansion rate (stiffness coefficient) of the object, and ρ is the density of the object (atmosphere).

In this case, assuming that air (atmosphere) is ideal air, ks=γp, γ is a heat capacity ratio (1.4 in the case of air), P is pressure, R is an ideal air constant, and T is absolute temperature [ K ]. Since temperatures are not included, they are all constant and therefore can be approximated.

The sound velocity in the atmosphere can be corrected for the temperature of the air, the atmospheric pressure, and the humidity and used for attenuation compensation.

Thereafter, under the control of the controller, a frequency domain signal may be acquired by performing signal transformation on a predetermined region of the corrected digital signal (2205).

In the frequency domain signal acquisition step 2205, a signal during a predetermined time period received after each period transmission delay of the sound signal (crosstalk signal of the transmitted sound signal is excluded from the corrected digital signal) may be frequency-converted by a signal converter to obtain a frequency domain signal.

Thereafter, under control of the controller, the frequency domain signal may be an input signal of a neural network, and features of the input signal may be extracted and classified based on the trained road classification model to determine a type of road surface (2206).

The neural network may include a Deep Neural Network (DNN) of a multi-layer perceptron algorithm that includes an input layer, a plurality of hidden layers, and an output layer. In this case, the output layer may output a probability value for each road surface type trained, and the neural network may determine and output the road surface type with the highest probability using softmax.

On the other hand, the structure of the neural network is not limited to the above DNN.

The neural network may receive the atmospheric information (temperature, humidity, and barometric pressure information) and may use it as an input to the input layer.

In addition, the neural network may frequency-convert the sound signal transmitted to the road surface, and may use it as an input to the input layer.

On the other hand, instead of acquiring the frequency domain signal in the frequency domain signal acquisition step and inputting the frequency domain signal to the artificial neural network, the artificial neural network may receive the digital signal after the atmospheric attenuation amount-reduction correction and perform a plurality of convolution operations under the control of the controller.

In the convolution operation step, a plurality of 1D convolution operations may be performed on the digital signal having the corrected air attenuation amount, and batch normalization, reLU function, and MaxPooling function may be performed on each convolution operation, and the output of the last convolution operation may be flattening data.

Thereafter, under the control of the controller, the characteristics of the convolution signal (flattening data) may be extracted and classified based on the road surface classification model trained by the artificial neural network to determine the type of road surface.

In the case of performing convolution, the artificial neural network may include a Deep Convolutional Neural Network (DCNN) of a multi-layer perceptron algorithm, which further includes a convolution performing unit that receives the digital signal and performs a plurality of convolution operations. The output layer may output a probability value for each road surface type trained, and the artificial neural network may determine and output the road surface type with the highest probability using softmax.

On the other hand, the structure of the artificial neural network is not limited to the DCNN described above.

The artificial neural network may receive the atmospheric information (temperature, humidity and barometric pressure information) and may use it as an input to the convolution execution unit.

In addition, the artificial neural network may also use a sound signal transmitted to the road surface as an input to the convolution performing unit.

Hereinafter, a detailed embodiment of installation and operation of the road surface classification device in the road infrastructure according to an embodiment of the present disclosure will be described in more detail.

Fig. 23 is a diagram for explaining a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure. Fig. 23 is a specific embodiment of the road infrastructure shown in fig. 2.

As shown in fig. 23, structure 2301 is located on or near road 2300, and an acoustic sensor 2310 and a visual sensor 2320 are disposed in structure 2301.

The sound sensor 2310 may be disposed in the structure 2301 so as to be located on and perpendicular to the road surface of the vehicle in the road 2300, and the visual sensor 2320 may be installed in the structure 2301 so as to photograph the entire road area.

On the other hand, in fig. 23, a communication unit 2350 for transmitting data obtained from the acoustic sensor 2310 and the visual sensor 2320 to a controller (not shown) is shown.

With the development of artificial neural networks, solutions in which the visual sensor 2320 is combined with artificial intelligence models are popular throughout the entire industrial field, and the visual sensor 2320 is one of the mainstream technologies in the fields of object recognition, detection, and segmentation. As artificial intelligence technology evolves, algorithms are implemented that enable the vision sensor 2320 to operate in a manner similar to the way a person intuitively recognizes objects from a photograph (image) and distinguishes regions.

On the other hand, by performing waveform analysis on a signal reflected after the surface of the object is identified using sound impact, an object can be identified using the sound sensor 2310, and a reflected wave is determined according to the acoustic impedance or surface roughness of the surface of the object realizing reflection. That is, if the sound sensor 2310 uses a wide range of sound spectrum, the sound sensor 2310 enhances external noise and can recognize black ice of a road surface.

Embodiments include a method of accurately recognizing a wide range of road surfaces by disclosing a visual sensor capable of intuitively recognizing a wide area by fusing and a sound sensor technology capable of accurately recognizing a target without being affected by a light source by using physical characteristics of a target object.

As shown in fig. 24, a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure may include a sound sensor 2410, a visual sensor 2420, an artificial neural network 2430, a segmentation processing unit 2440, and a controller 2470.

The sound sensor 2410 may transmit a sound signal to a predetermined point for road condition monitoring and may receive a reflected signal.

The vision sensor 2420 may acquire an image of a road surface including a predetermined point.

The artificial neural network 2430 may classify the road surface state of the predetermined point based on the learned road state classification model by using the reflected signal acquired by the sound sensor 2410 as an input signal. The road surface condition may include dry road, water, black ice and snow.

The segmentation processing unit (2440) may divide an image acquired by the vision sensor (2420) as an input signal into a plurality of differentiated segmentation regions based on the segmentation model.

The controller (2470) may control operations of the sound sensor (2410), the visual sensor (2420), the artificial neural network (2430), and the segmentation processing unit (2440), and determine the road surface state of the corresponding road by fusing the road surface state of a predetermined point output from the artificial neural network (2430) and a plurality of segmentation areas output from the segmentation processing unit (2440).

The process by which the controller (2470) determines the road surface state of the corresponding road will be described in detail with reference to fig. 27.

The controller (2470) may calculate a segmented region including a point at which the acoustic sensor contacts the ground. However, it is preferable that the touchdown point (sensing area) of the sound sensor is set to a normal icing section in the road surface when the system according to the present disclosure is installed, and the position of the sensing area may be known in the system.

Finally, the sound sensor may output image information by assigning a classification category (road surface type) of waveform data to a divided area including a point (sensing area) contacting the ground.

Fig. 25 is a diagram illustrating an example of identifying a uniform road surface state in a road state monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Referring to fig. 25, (a) is a photographed image of the visual sensor (2420), and the position of the sensing region (predetermined region) (2500) of the sound sensor (2410) is displayed. (b) The display of the divided areas by dividing the photographed image of (a) is shown. (c) The road surface type is classified based on a trained road surface classification model (artificial intelligence model), and the road surface state of the sensing region (2500) is sensed. (d) An image of a black ice region is finally displayed by finding a divided region including the region (2500) sensed in (c) from among the divided regions of (b) is shown.

Here, (b) shows that the entire area of the road is divided into one divided area, and the sensing area (2500) of the road surface sensed by the sound sensor (2410) is sensed as black ice, and thus the result of (d) can be finally outputted.

Fig. 26 is a diagram illustrating an example of identifying an uneven road surface condition in a road condition monitoring system that includes a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Referring to fig. 26, (a) is a captured image of the visual sensor (2420), and the position of the sensing region (predetermined region) (2600) of the sound sensor (2410) is displayed. (b) The display of the divided areas by dividing the photographed image of (a) is shown. (c) The road surface type is classified based on a trained road surface classification model (artificial intelligence model), and the road surface state of the sensing region (300) is sensed. (d) An image in which a black ice region is finally displayed by finding the divided region of (b) is shown.

Here, (b) may be divided into a plurality of divided regions including a road wet region and a dry region, and since the sensing region 2600 detected by the sound sensor 2410 is detected as black ice, the result as shown in (d) may be finally output.

In other words, the present disclosure may include a technique capable of reliably determining, by the visual sensor, which part/region of the road surface image obtained by the visual sensor corresponds to the problem of the road surface information correctly recognized by the acoustic sensor.

In the present disclosure, the operation of the road surface detection algorithm may be performed by periodic (in minutes, seconds) or asynchronous requests, and the sensing of the road surface slip risk may be performed by acquiring data by the sound sensor and the vision sensor, sensing the type (state) of the road surface based on the data acquired by the sound sensor, and detecting the region including the sensing portion of the sound sensor by image segmentation in the image acquired by the vision sensor.

That is, when the danger of black ice is sensed in the road surface, the danger information may be combined in the image divided into the areas as the division result, and the danger information may be used in the form of transmitting a notification of the road surface danger section to the administrator (management server).

Fig. 27 is a diagram for explaining a method of finding a divided area of a sensing area of a sound sensor in a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure.

The controller may determine the road surface state of the road by fusing the road surface state of the predetermined point output from the artificial neural network and the plurality of divided regions output from the division processing unit 2440.

The controller calculates a position of a midpoint of each of the divided regions, calculates a linear equation of a plurality of line segments included in each of the divided regions (the plurality of line segments forming each of the divided regions), determines a first positive-negative relationship between the midpoint of the corresponding region and each of the plurality of line segments using the linear equation for each of the divided regions, determines a second positive-negative relationship between a predetermined point and each of the plurality of line segments using the linear equation for each of the divided regions, and determines a divided region in which the second positive-negative relationship and the first positive-negative relationship match each other as a region including the predetermined point.

Referring to fig. 27, it is assumed that an input RGB image is divided into a plurality of regions after division, and wherein an a region and a B region are divided as shown.

The midpoint of the "a region" is denoted by "2701", the midpoint of the "B region" is denoted by "2702", and the sensing region (predetermined point) of the sound sensor is denoted by "2700".

In fig. 27, positive and negative relations are defined as points (0, 0) with respect to the upper left corner of the image, the right side and the lower side being (+) directions, and the left side and the upper side being (-) directions.

The a region is formed into a pentagon, and is constituted by line segments 14, 45, 56, 67, and 71.

The positive and negative relationship between the midpoint 2701 of the a region and the line segments 14, 45, 56, 67, 71 of the a region becomes (-), (+) and (+), respectively.

The B region is formed in a quadrangle, and includes line segments 12, 23, 34, and 41.

The positive and negative relationship between the midpoint 2702 of the B region and the line segments 12, 23, 34, and 41 of the B region becomes (+), (-), and (+), respectively.

In this case, the positive and negative relations between the detection region (predetermined point) of the sound sensor 2700 and the line segments 14, 45, 56, 67, and 71 of the a region become (+), (-), (+) and (+), respectively, and the positive and negative relations between the line segments 12, 23, 34, and 41 of the B region become (+), (-) and (+), respectively.

Therefore, a detection area (predetermined point) of the sound sensor 2700 is included in the B area.

Fig. 28 is a diagram for explaining an example of an artificial neural network of a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure.

Referring to fig. 28, the artificial neural network may be formed of an artificial intelligence model implemented by any one of 1D CNN (conventional neural network) or ANN (artificial neural network).

The input of the artificial neural network may be a reflected signal received through the acoustic sensor, and the output thereof may be a road surface type of a predetermined detection area of the acoustic sensor.

Fig. 29 is a diagram for explaining an example of a segmentation processing unit of a road condition monitoring system including a vision sensor and a sound sensor according to an embodiment of the present disclosure.

Referring to fig. 29, the segmentation processing unit may be formed of an image segmentation model based on a Conventional Neural Network (CNN) implemented by an auto encoder or U-Net.

The input of the segmentation processing unit is an RGB image acquired by a visual sensor, and the output thereof is a segmented image of the region differentiated in the image.

Fig. 30 is a flow chart of a monitoring method in a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure.

First, for road condition monitoring, the sound sensor may receive the reflected signal 3010 after transmitting the sound signal to a predetermined point.

Then, the road surface condition of the predetermined point may be classified based on a trained road surface classification model having the reflected signal acquired by the sound sensor as an input signal (3020).

On the other hand, when the acoustic sensor transmits an acoustic signal and receives a reflected signal, the visual sensor may acquire an image of a road surface including a predetermined point (3030).

Then, the image acquired by the vision sensor may be divided into a plurality of differentiated segmentation areas based on the segmentation model (3040).

The road surface condition may then be analyzed by combining the road surface condition at the predetermined point and the plurality of distinguished segmented regions (3050).

Thereafter, the road surface state of the corresponding road may be determined according to step 3050 of analysis by merging (3060).

Thereafter, it may be determined whether a hazard is detected in the road surface state of the corresponding road (3070).

As a result of the determining danger step 3070, there may be no danger detected in the road surface state of the corresponding road, and thus steps "3010" and "3030" may be periodically performed.

On the other hand, as a result of the determining danger step 3070, when a danger is detected in the road surface state of the corresponding road, a signal for notifying the management server of the danger may be transmitted (3080).

On the other hand, in the danger notifying step (3080), the danger area may be displayed on an image including a plurality of distinguished divided areas and transmitted to the management server.

The step 3050 of analyzing by merging may include performing the following steps.

First, in the divided image, the position of the midpoint of each divided region is calculated (3051).

Thereafter, a linear equation is calculated for a plurality of line segments included in each divided region (a plurality of line segments forming each divided region) (3052).

Thereafter, for each segmented region, a first positive-negative relationship of each of the plurality of line segments to a midpoint of the respective region is determined using a linear equation (3053).

Thereafter, for each segmented region, a second positive-negative relationship of each of the plurality of line segments to the predetermined point is determined using a linear equation (3054).

Thereafter, a divided region in which the second positive-negative relationship and the first positive-negative relationship match each other is determined as a region including a predetermined point (3055).

Fig. 32 is a construction diagram of a road condition monitoring system including a visual sensor and a sound sensor according to another embodiment of the present disclosure.

As shown in fig. 32, a road condition monitoring system including a visual sensor and a sound sensor according to an embodiment of the present disclosure may include a sound sensor 3210, a visual sensor 3220, a first feature extractor 3281, a second feature extractor 3282, a combined artificial neural network (joint classifier) 3290, and a controller 3270.

The sound sensor 3210 may transmit a sound signal to a predetermined point for road condition monitoring and receive a reflected signal.

The vision sensor 3220 may acquire an image of a road surface including a predetermined point.

The first feature extractor 3281 may extract first features from the reflected signal acquired by the sound sensor 3210.

The second feature extractor 3282 may extract second features from the image acquired by the vision sensor 3220.

The combined artificial network 3290 may classify the road surface states of the respective roads based on a learned road surface data joint classification model based on the input of the signals acquired by the sound sensor 3210 and the images acquired by the visual sensor 3220. Road conditions may include dry, watery, black ice (ice) and snow.

The controller 3270 may control operations of the sound sensor 3210, the visual sensor 3220, the first feature extractor 3281, the second feature extractor 3282, and the combined artificial neural network 3290.

In the combined artificial neural network 3290, the first features extracted from the reflected signal and the second features extracted from the image may be trained and classified by a classification model (data joint classification model) that weights each separately.

The first feature extracted from the reflected signal and the second feature extracted from the image can learn the object (road surface type) classification by combining the image data and the sound data using the correlation (correspondence). Further, training may also be performed by analyzing the characteristics of the data to adjust the weights and effects of the image data-based classifier and the sound data-based classifier to make a final decision (prediction).

Fig. 33 is a flow chart of another embodiment of a monitoring method in a road condition monitoring system including a visual sensor and a sound sensor according to the present disclosure.

First, for road condition monitoring, the sound sensor may transmit a sound signal to a predetermined point and then receive a reflected signal (3310).

Thereafter, a first characteristic of the reflected signal may be extracted (3320).

On the other hand, when the sound sensor emits sound and receives a reflected signal, the vision sensor may acquire an image of a road surface including a predetermined point (3330).

Thereafter, a second feature of the image may be extracted (3340).

Thereafter, the road surface state of the corresponding road may be analyzed based on a classification model trained by combining the first features extracted from the reflected signal and the second features extracted from the image (3350).

Here, the first feature extracted from the reflected signal and the second feature extracted from the image may be trained and classified by a classification model that weights each separately.

Thereafter, the road surface state of the corresponding road may be determined according to the analyzing step 3350 of the road surface state (3360).

Thereafter, it may be determined whether a hazard is detected in the road surface state of the corresponding road (3370).

As a result of the determining danger step 3370, there may be no danger detected in the road surface state of the corresponding road, and thus the steps of "3310" and "3330" may be periodically performed.

On the other hand, as a result of the determining danger step 3370, a danger may be detected in the road surface state of the corresponding road, and thus the danger may be signaled to the management server (3380).

On the other hand, in the danger notifying step 3380, the danger area may be displayed on an image including a plurality of different divided areas and may be transmitted to the management server.

Hereinafter, specific embodiments for controlling a heating wire device or a brine spraying device as an example of the road surface management device according to the embodiment will be described in detail.

Fig. 34 is a diagram for explaining the operation of the control system of the heating wire device according to an embodiment. Fig. 34 is a specific embodiment of the road infrastructure shown in fig. 2.

As shown in fig. 34, the control system of the heating wire apparatus of the road according to an embodiment of the present disclosure may include a structure 3401 located on the road, the structure 3401 may include a sound sensor 3410 and a communication unit 3420, and an automatic control box 3460 may be controlled based on sensing data of the sound sensor 3410 under the control of a control server 3440, the automatic control box 3460 controlling the heating wire 3470 from the anti-freezing apparatus 3400.

The acoustic sensor 3410 may be mounted on the structure 3401 so as to be located on the road of the vehicle and perpendicular to the road surface, but is not limited thereto.

The structure 3401 refers to an object, such as a street lamp, on which the sound sensor 3410 may be mounted on a road.

The sound sensor 3410 may transmit a sound signal to a predetermined point to sense a road condition and then receive a reflected signal.

On the other hand, the communication unit 3420 may transmit the data acquired through the sound sensor 3410 to the control server 3440.

Further, when the control server 3440 senses whether or not the sound sensor 3410 is malfunctioning (or abnormal), the control server 3440 may transmit a notification to the management terminal 3450.

In the antifreeze apparatus, in the wire method, the operation time of the wire may exceed the required time, thereby causing a fire in asphalt, and thus the amount of operation that the wire should perform is important.

In the present disclosure, the antifreeze device may be precisely controlled by sensing a change in road surface temperature due to heating of the heating wire by the sound sensor.

Specifically, an artificial intelligence model that learns based on sound sensing data accumulated in a road surface environment where various temperatures are applied may be generated, and the operation of the heating wire device may be automatically controlled by sensing a temperature change of the road surface by analyzing waveforms of sound sensors acquired based on the artificial intelligence model.

For example, the heating wire may be started after grasping whether the road surface is dry or frozen by waveform analysis of the acoustic sensor, and the operation of the heating wire may be set to stop when the road surface temperature output from the acoustic sensor is maintained above 4 degrees celsius, that is, when the temperature is higher than the freezing point of water.

Further, according to the present disclosure, the amount of brine sprayed onto the road surface during operation of the brine spraying device may be sensed.

Specifically, an artificial intelligence model that learns based on sound sensing data accumulated in various distributed road surface environments may be generated, and waveforms of sound sensors acquired based on the artificial intelligence model may be analyzed to determine the degree of dispersion (distribution) of brine sprayed onto the road surface (degree of spraying). If the brine is distributed within a predetermined range of the road surface, it may be set to stop brine spraying.

The present disclosure relates to a road surface sensing and interlocking control technology in which an installed/operated road heating wire device or a brine spray device can be operated in time, and the operation of the heating wire device or the brine spray device can be precisely controlled by determining road surface state recognition and melting conditions based on a sound sensor, instead of a conventional method of obtaining road surface information by using a temperature/humidity sensor.

It can be embedded into the existing snow removing apparatus monitoring/control system without significant modification, and can provide services based on more accurate road hazard notification than the existing snow removing system to improve the operating efficiency of the snow removing apparatus.

The algorithm for determining whether to perform snow removal may be built in the control server (service server) 3440, or may be built in a controller (MCU) provided together with the sound sensor 3410, and may be in the form of an automatic control box 3460 transmitted to the freezing point depressant 3400 through a communication unit.

The sensing of the road surface state is periodically repeated until the acoustic sensor needs to be restored due to the malfunction of the acoustic sensor. When the sound sensor 3410 acquires the reflected wave (sensor value) and then transmits it to the control server 3440 (service server) through the communication unit, the control server 3440 analyzes the reflected wave using the artificial intelligence model based on big data to determine whether snow removal is currently required and controls whether the corresponding anti-freezing device is operated when the sensor state is normal.

On the other hand, whether normal or abnormal is determined by the received sensor value, when the sensor state is abnormal, an operation stop command is transmitted to the automatic control box 3460 of the anti-freeze apparatus 3400, a push alert is transmitted to the management terminal 3450 for the occurrence of abnormality, and the operation stop history caused by the failure is transmitted to the control server (not shown).

Fig. 35 is a configuration diagram of a control system of a road freeze-proofing device according to an embodiment of the present disclosure. The control system of fig. 35 is a specific example of a road surface management device according to an embodiment of the present disclosure.

As shown in fig. 35, a control system of a road freeze protection device according to an embodiment of the present disclosure may include a sound sensor 3510, a control server 3540, a communication unit 3520, and a freeze protection device 3500.

The sound sensor 3510 may transmit a sound signal to a predetermined point to sense a road surface state and then receive a reflected signal.

The control server 3540 may sense road surface state data of a predetermined point based on the learned artificial intelligence analysis model by using the reflected signal acquired by the sound sensor 3510 as an input signal, and generate a signal for controlling whether the anti-freeze 3500 operates according to the road surface state data of the predetermined point.

The road surface state data may include weather conditions, road surface type, road surface temperature, and brine spray amount (degree of spraying, degree of distribution).

The communication unit 3520 may transmit the reflected signal acquired by the sound sensor 3510 to the control server 3540.

The freezing prevention apparatus 3500 is controlled by the control server 3540 to perform an operation for preventing road freezing. The anti-freeze apparatus 3500 may include at least one of a heating wire apparatus or a brine spray apparatus.

Specifically, when the weather condition is "rain" or "snow", the classified road surface type is "wet road surface" or "snow road surface" or "ice road surface", and the sensed road surface temperature is less than 4 degrees celsius, the control server 3540 may generate a control signal for operating the heating wire device, and when the sensed road surface temperature is 4 degrees celsius or more after the heating wire device is operated, the control server 3540 generates a control signal for stopping the operation of the heating wire device.

On the other hand, when the weather condition is "rain" or "snow", the classified road type is "wet road" or "snow road" or "ice road", and the sensed road temperature is less than 4 degrees celsius, the control server 3540 may generate a control signal for operating the brine spray, and when the sensed brine spray after the brine spray is operated is 80% or more, the control server 3540 generates a control signal for stopping the brine spray operation.

On the other hand, when sensing that the state of the sound sensor 3510 is abnormal, the control server 3540 may transmit a notification message to the management terminal 3550, and may transmit a state notification signal of the sound sensor 3510 to the management server 3580.

As shown in fig. 36, the control system of the road freeze-proofing device according to an embodiment of the present disclosure includes a sound sensor 3610, a controller 3630, a communication unit 3620, and a freeze-proofing device 3600.

The sound sensor 3610 may transmit a sound signal to a predetermined point to sense a road condition and receive a reflected signal.

The controller 3630 may sense road surface state data at a predetermined point based on an artificial intelligence analysis model learned by using a reflected signal acquired by the sound sensor 3610 as an input signal and generate a signal for controlling whether the freezing apparatus 3600 operates according to the road surface state data at the predetermined point.

The communication unit 3620 may transmit a control signal generated by the controller 3630 to the freezing apparatus 3600.

The freezing prevention device 3600 is controlled by the controller 3630 to perform an operation for preventing ice formation on a road. The freeze guard 3600 may include at least one of a heater wire arrangement or a brine spray arrangement.

Specifically, when the weather condition is "rain" or "snow", the classified road surface type is "wet" or "snow covered" or "frozen", and the sensed road surface temperature is less than 4 degrees celsius, the controller 3630 may generate a control signal for operating the heating wire device, and when the sensed road surface temperature is 4 degrees celsius or more after the operation of the heating wire device, the controller 3630 generates a control signal for stopping the operation of the heating wire device.

On the other hand, when the weather condition is "rain" or "snow", the classified road surface type is "wet" or "snow covered" or "frozen", and the sensed road surface temperature is less than 4 degrees celsius, the controller 3630 may generate a control signal for operating the brine spray, and when the sensed brine spray after the brine spray operation is 80% or more, the controller 3630 may generate a control signal for stopping the brine spray operation.

On the other hand, when sensing that the state of the sound sensor 3610 is abnormal, the controller 3630 may transmit a notification message to the management terminal 3650 and may transmit a state notification signal of the sound sensor 361 to the control server 3680.

The artificial intelligence analysis model may include a weather condition classification model for classifying weather conditions based on reflected signals acquired by the sound sensor and a road surface type classification model for classifying road surface types based on reflected signals acquired by the sound sensor.

When the antifreeze device is a heater wire device, the artificial intelligence analysis model may further include a road surface temperature regression model that learns the reflected signal acquired by the sound sensor together with the road surface temperature and outputs a corresponding road surface temperature based on the reflected signal acquired by the sound sensor.

When the antifreeze device is a brine spray device, the artificial intelligence analysis model may further include a road surface temperature regression model that outputs a corresponding road surface temperature by learning together the reflected signal acquired by the sound sensor and the road surface temperature based on the reflected signal acquired by the sound sensor, and a brine spray amount regression model that outputs a corresponding brine spray amount (distribution degree) by learning together the reflected signal acquired by the sound sensor and the amount of brine spray based on the reflected signal acquired by the sound sensor.

That is, the artificial intelligence analysis model according to the present disclosure basically includes a weather condition classification model, a road surface type classification model, and a road surface temperature regression model, and when the antifreeze device includes a brine spray, the artificial intelligence analysis model may further include a brine spray regression model.

Fig. 37A shows the structure of the road surface type classification model, and the road surface type classification model is configured by sampling the acquired signal (reflected signal) of the sound sensor a total of T times within a predetermined time and learning the corresponding road surface type information together.

For example, 1000 pieces of data (x 1, x2, …, x 1000) sampled in units of 1ms within 1 second and corresponding road surface type information are learned.

As shown in fig. 37A, the signal (reflected signal) of the acquired sound sensor may be input, and the corresponding road surface type may be classified into categories such as dry, wet, frozen, and snowy.

On the other hand, although not shown in the drawings, a weather state classification model that classifies weather conditions based on reflected signals acquired by the sound sensors may be further included, and the weather state classification model also learns the acquired signals of the sound sensors together with weather information.

Fig. 37B is a diagram for explaining a road surface temperature regression model that outputs a corresponding road surface temperature based on the reflected signal acquired by the sound sensor to control the heating device, and fig. 37C is a diagram for explaining a brine spray amount regression model that outputs a corresponding brine distribution amount (distribution degree) (%) based on the reflected signal acquired by the sound sensor to control the brine control device.

In fig. 37B, the relationship between the acquired data X of the sound sensor and the road surface temperature is not a two-dimensional (planar) map, and X is a data set formed by the concept of a hyperplane, which is a set of various values.

Thus, the trained road surface temperature regression model outputs a corresponding road surface temperature based on the reflected signal acquired by the acoustic sensor.

Similar to fig. 37B, as shown in fig. 37C, the relationship between the acquired data X of the sound sensor and the amount of sprayed saline (degree of distribution) is not a two-dimensional (plane) graph, and X is a data set formed by the concept of a hyperplane, which is a set of various values.

Thus, the trained brine spray regression model outputs the corresponding brine spray (degree of distribution) based on the reflected signal acquired by the sound sensor.

Fig. 38 is a flowchart of an embodiment of a control method of the road freeze-proofing device according to the present disclosure.

First, measurement data of a sound sensor is collected (3810).

An artificial intelligence analytical model is generated based on the collected data (3820).

The artificial intelligence analysis model generation step 3820 generates a weather condition classification model that classifies weather conditions based on the reflected signal acquired by the acoustic sensor, generates a road surface type classification model that classifies road surface types based on the reflected signal acquired by the acoustic sensor, and generates a road surface temperature regression model that outputs road surface temperature by learning the reflected signal and the road surface temperature together based on the reflected signal acquired by the acoustic sensor.

On the other hand, when the antifreeze apparatus is a brine spray apparatus, the artificial intelligence analysis model generation step 3820 learns the reflected signal acquired by the sound sensor and the brine spray amount (distribution degree) together to generate a brine spray amount regression model that outputs a corresponding spray amount (distribution degree) based on the reflected signal acquired by the sound sensor.

That is, the artificial intelligence analysis model according to the present disclosure basically generates and includes a weather state classification model, a road surface type classification model, and a road surface temperature regression model, and further generates a brine spray regression model when the antifreeze apparatus includes a brine spray.

Of course, the generated artificial intelligence analytical model should be installed on the control server or controller.

Thereafter, a sound signal is transmitted to a predetermined point to monitor the state of the road using the sound sensor, and then a reflected signal is received (3830).

Thereafter, the control server or controller senses the road surface state data at the predetermined point based on the artificial intelligence analysis model by using the reflected signal acquired by the sound sensor as an input signal.

The road surface state data sensing step 3840 senses weather conditions, road surface type, road surface temperature, and brine spray amount (distribution degree).

Thereafter, the control server or controller generates a control signal (3850) for controlling whether the antifreeze apparatus is operated based on the road surface state data.

Thereafter, when the control server or the controller detects that the state of the sound sensor is abnormal, a notification message is transmitted to the management terminal, and a state notification signal of the sound sensor is transmitted to the management server (3860).

Fig. 39 is a detailed flowchart of the control signal generation step 3850 of fig. 38 when the road freeze-protection device according to the present disclosure is a heater wire device.

When the anti-freeze device is a heater wire device, the control signal generation step 3850 first determines whether the sensed weather condition is "rain" or "snow" (3910).

As a result of the determining step 3910, if the weather condition is not "rain" or "snow", step "3840" is performed to detect the road surface state data.

On the other hand, as a result of the determining step 3910, if the weather condition is "rain" or "snow", it is determined whether the classified road surface type is "wet" or "snow covered" or "frozen" (3920).

As a result of the determining step 3920, if the weather condition is "rain" or "snow", and if the classified road surface type is not "wet" or "snow covered" or "frozen", then step "3840" is performed to detect the road surface state data.

On the other hand, as a result of the determining step 3920, if the weather condition is "rain" or "snow", and if the classified road surface type is "wet" or "snow covered" or "frozen", it is determined whether the road surface temperature is below 4 degrees celsius (3930).

As a result of the determining step 3930, if the road surface temperature is not lower than 4 degrees celsius, step S540 is performed to detect the road surface state data.

On the other hand, as a result of the determining step 3930, if the road surface temperature is lower than 4 degrees celsius, a control signal for operating the heating wire device is generated (3940).

Thereafter, a road surface state data sensing step 3940 is performed to detect road surface state data.

After the heater wire assembly is activated, it is determined whether the road surface temperature is greater than or equal to 4 degrees celsius (3950).

As a result of the determining step 3950, if the road surface temperature is not greater than or equal to 4 degrees celsius, step "S540" is performed to detect the road surface state data.

On the other hand, as a result of the determining step 3950, if the road surface temperature is greater than or equal to 4 degrees celsius, a control signal for stopping the operation of the heating wire device is generated (3960).

Thereafter, a road surface state data sensing step 3840 is performed to detect road surface state data.

In other words, when the weather condition is "rain" or "snow", the classified road surface type is "wet" or "snow covered" or "frozen", and the sensed road surface temperature is lower than 4 degrees celsius, the heating wire device is operated, and after the heating wire device is operated, when the sensed road surface temperature is 4 degrees celsius or more, the operation of the heating wire device is stopped.

Fig. 40 is a detailed flowchart of the control signal generating step 3850 of fig. 38 when the road freeze protection device according to the present disclosure is a brine spray.

When the antifreeze apparatus is a brine sprinkler, the control signal generation step 3850 first determines whether the sensed weather condition is "rain" or "snow" (4010).

As a result of the determining step 4010, if the weather condition is not "rain" or "snow", step S540 is performed to detect the road surface state data.

On the other hand, as a result of the determining step 4010, if the weather condition is "rain" or "snow", it is determined whether the classified road surface type is "wet" or "snow covered" or "frozen" (4020).

As a result of the determining step 4020, if the weather condition is "rain" or "snow" and the classified road surface type is not "wet" or "snow covered" or "frozen", then step "3840" is performed to detect the road surface state data.

On the other hand, as a result of the determining step 4020, if the weather condition is "rain" or "snow", and the classified road surface type is "wet" or "snow covered" or "frozen", it is determined whether the road surface temperature is lower than 4 degrees celsius (4030).

As a result of the determining step 4030, if the road surface temperature is not lower than 4 degrees celsius, step "3840" is performed to detect the road surface state data.

On the other hand, as a result of the determining step 4030, if the road surface temperature is lower than 4 degrees celsius, a control signal (4040) for operating the brine spray is generated.

Thereafter, step "3840" is performed to detect the road surface state data.

After the brine spray is operated, it is determined whether the brine spray (degree of spraying) is 80% or more (4050).

As a result of the determination step 4050, if the brine spray amount (degree of injection) is not 80% or more, step "3840" is performed to detect the road surface state data.

On the other hand, as a result of the determination step 4050, if the brine spray amount (degree of spraying) is 80% or more, a control signal for stopping the operation of the brine spray is generated (4060).

Thereafter, step "3840" is performed to detect the road surface state data.

In other words, when the weather condition is "rain" or "snow", the classified road surface type is "wet road surface" or "snow road surface" or "ice road surface", and the sensed road surface temperature is lower than 4 degrees celsius, the brine spray device is operated, and after the brine spray device is operated, when the sensed brine spray amount is 80% or more, the operation of the brine spray device is stopped.

On the other hand, the brine spray amount (spray degree, distribution degree) for stopping the operation of the brine spray device is 80% or more, but is not limited thereto.

On the other hand, in order to control the operation of the brine spray, all the road surface temperature regression models and the brine spray regression models are used together, but only the brine spray regression models may be used to control the operation of the brine spray.

Although the above examples include the heating wire device or the brine spray device as the road freeze protection device, the present disclosure is not limited thereto, and a system including the heating wire device and the brine spray device may also be controlled together.

In the above description, the method according to an embodiment of the present disclosure may be implemented by a computer-readable recording medium having a program for implementing the method stored thereon and/or a program for implementing the method stored in the computer-readable recording medium.

That is, it will be readily understood by those skilled in the art that the program of instructions for implementing the method according to an embodiment of the present disclosure is tangibly embodied so that the program can be included in a recording medium readable by a computer. In other words, it may be implemented in the form of program instructions capable of being executed by various computer devices and recorded in a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

When the methods of the present disclosure are implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. One or more programs stored in the computer-readable storage medium are configured to be executable (configured for execution) by one or more processors in the electronic device. The one or more programs include instructions that cause the electronic device to perform a method according to embodiments described in the claims or specification of the present disclosure.

Such programs (software modules, software) may be stored in non-volatile memory including random access memory, flash memory, read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), magnetic disk storage devices, compact disk read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage devices, and magnetic cassettes. Alternatively, the memory may be stored in a memory configured by a combination of some or all of them. Further, each configuration memory may be included in a plurality of memories.

Further, the program may be stored in a connectable storage device that can be accessed through a communication network such as the internet, an intranet, a Local Area Network (LAN), a wide area network (WLAN), or a SAN (storage area network), or a combination thereof. Such a storage device may access devices that perform embodiments of the present disclosure through an external port. Further, a separate storage device on the communication network may access the device performing embodiments of the present disclosure.

Hereinafter, a method for installing a road infrastructure in a road is exemplarily described. More specifically, in the following description, the structure is the structure of fig. 2, 23, and 34, and relates to the structure of the present disclosure and the installation method of the structure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to fig. 41 to 49 showing embodiments of the present disclosure.

According to an embodiment of the present disclosure, structure 4100, which is a main component of the road infrastructure sensor construction structure of the present disclosure, includes: a vertical frame 4110 located on a road or road edge; and a horizontal frame 4120 installed at an upper portion of the vertical frame 4110 in the road width direction, and the structure 4100 may be, for example, a street lampShaped or mounted to the general lighting panelThe structure of the bridge (hypass) ic structure, and if the horizontal frame 4120 is located at the upper portion of the road, any structure may be provided, and the sound sensor unit 4200 described in detail later may be installed below the horizontal frame 4120 and at the upper portion of the road. As described above, the position of the sound sensor unit of the present disclosure is exemplary and is not limited to the position depicted in the present disclosure.

According to an embodiment of the present disclosure, the sound sensor unit 4200, which is a main component of the road infrastructure sensor construction structure of the present disclosure, is installed below the horizontal frame 4120 of the structure 4100, is installed to be located at an upper portion of a road, emits sound signals to a road surface of the road, receives sound reflected from the road surface of the road and generates sound information, and transmits the received sound information to the controller 4300, which will be described in detail later, such that the sound information is converted into frequency and a road surface state can be recognized by the controller 4300.

The sound sensor unit 4200 of the present disclosure may include a transmitter that transmits a sound signal to a road surface of a road, receives a reflected sound signal, and outputs the transmitted signal under the control of the controller 4300, and a receiver that receives a reflected signal reflected from the road surface of the road.

The emitter and the receiver are positioned such that sound is emitted and reflected in a straight line form, and a sound sensor 4240 to be described below is shown as being configured in a state where the emitter and the receiver are integrated.

On the other hand, in the method of using the voice recognition of the road surface state, since the transmitter and the receiver are fixedly installed at an angle, only the state of a very narrow area of the road surface can be recognized, and in the present disclosure, by an embodiment including a plurality of the voice sensor units 4200 and another embodiment including the first transceiving member 4250 and the second transceiving member 4260 which can mutually receive the reflected voice signal, the road surface state can be understood to be wider than the related art, thereby enabling the state of the road to be estimated with high reliability.

Specifically, the sound sensor unit 4200 of an embodiment is mounted below the horizontal frame 4120 of the structure 4100, and a plurality of sound sensor units can be mounted in the width direction of the road, so that reliability capable of recognizing the road surface state can be improved.

In this case, the plurality of sensor units 4200 mounted in an embodiment is characterized in that the emission times of sounds are set so that the arrival times of sounds radiated and reflected from the road surface are all the same, in order that the characteristics of sounds in a normal road surface state as an initial standard can be extracted and compared relatively similarly by setting the emission times of sounds, whereby recognition of the road surface state can be rapidly achieved.

Further, the reason why the emission time of the sound is set so that the arrival time is the same is that when the sound uses the emission time and the arrival time to generate sound information, since the speeds of the sound are the same when the plurality of sound sensor units 4200 of the same device are installed, if the emission time is the same, since the arrival times of the sound reflected from the road surface are different based on the variation of the road surface height, it is necessary to analyze the sound information generated by each sound sensor unit 4200, but in order to easily generate sound information even in the case of having the arrival time of the sound, when the emission time of the sound of each sound sensor unit 4200 is adjusted so that the arrival times of the sound are the same based on the variation of the road surface height, the difference of the sound information having different arrival times can be quickly determined, so that the recognition of the road surface state can be easily achieved.

For example, when the emission times of the sounds of the plurality of sound sensor units 4200 are set to be the same in accordance with the road surface height, the arrival times of the sounds are the same when the road surface is in a general state, and the arrival times of the sounds are different from the predetermined arrival times of the sounds when the road surface state is in a state such as a damaged or black ice, so that the recognition of the state of the entire road surface can be easily achieved, but when the arrival times of the sounds of some of the sound sensor units 4200 are different from the predetermined arrival times of the sounds, it can be checked more quickly that some of the states of the entire road surface are different.

Further, an embodiment of the plurality of sound sensor units 4200 is characterized in that when the time at which sound is emitted from any one of the sound sensor units 4200 and reflected by the road surface and returned (i.e., sound flight period) is t, n sensors sequentially emit a total of n or more sounds to the road surface and receive the reflected sound for the remaining t/2 hours, thereby sampling road surface information or traffic volume information n or more times within a predetermined period.

That is, an embodiment of the sound sensor unit 4200 achieves the effect of sampling sound information such as road surface information and traffic information and generating the sound information by using a plurality of sound sensor units 4200, so that the state of the road surface can be obtained more reliably. In this case, it is apparent that another embodiment of the sound sensor unit 4200, which will be described in detail later, also samples by sound and generates sound information such as road surface information and traffic information.

Furthermore, an embodiment 4200 of the sound sensor unit includes: a coupling assembly 4210, the upper portion of which is coupled to the lower portion of the horizontal frame 4120 of the structure 4100; a connection lever 4220 whose upper portion is hinged to a lower portion of the coupling assembly 4210 so as to be rotatable in front-rear and left-right directions; a main body 4230 coupled to a lower end of the connection rod 4220; and a sound sensor 4240 mounted at a lower portion of the main body 4230. In this case, the front-rear direction refers to the longitudinal direction of the road, the left-right direction refers to the width direction of the road, and the below-mentioned front-rear and left-right directions (lateral directions) also refer to such directions.

The coupling assembly 4210 includes a lower receiving groove 4212 formed at a lower portion and a ball bearing installed in the lower receiving groove 4212, and the connection rod 4220 includes an upper ball 4224, the upper ball 4224 being inserted into and coupled to an inner side of the lower receiving groove 4212 of the coupling assembly (4210) at an upper end thereof such that it is coupled in a ball hinge manner to be rotatable in front-rear and left-right directions.

In this case, it is apparent that the lower receiving groove 4212 is formed in a shape corresponding to the upper ball 4224 of the connection rod 4220, and the connection rod 4220 is coupled to the lower portion of the coupling assembly 4210 in a ball hinge structure rotatable in front-rear and left-right directions, thereby preventing sloshing when vibration is generated in the structure 4100 due to vehicle traffic and external vibration.

That is, the body 4230 coupled to the lower end of the connection rod 4220 and the sound sensor 4240 mounted at the lower portion of the body 4230 may collectively prevent shaking of the connection rod 4220 to stably radiate sound set by the sound sensor 4240 to be radiated to the road surface.

As described above, the main body 4230 allows the sound sensor 4200 just exemplified and another embodiment of the sound sensor 4200 described in detail later to be stably installed and connected to the controller 4300 described in detail later to be controlled by the controller 4300, and the sound sensor 4240 generates sound information by radiating sound to a road surface and then receiving reflected sound and transmits the generated sound information to the controller 4300 as described below.

Next, another embodiment of the sound sensor unit 4200 includes: a first transceiving member 4250 installed at one side of a lower portion of the horizontal frame 4120 of the structure 4100 to radiate sound to or receive sound from a road surface; and a second transceiving member 4260 installed at the other side of the lower portion of the horizontal frame 4120 of the structure 4100 to radiate sound to or receive sound from the road surface.

In this case, the first transceiving member 4250 is installed at an angle capable of receiving sound radiated and reflected from the second transceiving member 4260 to the road, and the second transceiving member 4260 is installed at an angle capable of receiving sound radiated and reflected from the first transceiving member 4250 to the road.

In other words, the sound sensor unit 4200 according to another embodiment allows any one of the first and second transceiving members 4250 and 4260 to emit sound to a road surface and receives the reflected sound through the second transceiving member 4260 or the first transceiving member, thereby realizing an effect of smoothly grasping a road surface state by using the other sound emitting unit when the sound emitting unit of any one of the first and second transceiving members 4250 and 4260 fails.

Further, the sound sensor unit 4200 according to another embodiment adjusts an angle of either the first transceiving member 4250 or the second transceiving member 4260 and allows the second transceiving member 4260 or the first transceiving member 4250 to move to receive sound according to the adjusted angle, thereby grasping a wider range of road surface states than the related art.

In more detail, in embodiment 1 of other embodiments, the horizontal frame 4120 of the structure 4100 for mounting the sound sensor unit 4200 includes a first rail 4122 mounted in front of a lower portion in the longitudinal direction and a rotation motor 4126 coupled to the first rail 4122 and controlled by the controller 4300 to rotate the first rail 4122.

In embodiment 1 of other embodiments, the sound sensor unit 4200 (i.e., the first transceiving member 4250 and the second transceiving member 4260) includes: the coupling assembly 4210 installed at one side and the other side of the first rail 4122 and moved in opposite directions according to the rotation of the first rail 4121; a connection lever 4220 coupled to a lower portion of the coupling assembly 4210 such that an upper portion can rotate in front-rear and left-right directions; a main body 4230 coupled to a lower end of the connection rod 4220; a driving motor 4270 installed at a lower portion of the main body 4230 and controlled by the controller 4300; and a sound sensor 4240 installed at a lower portion of the driving motor 4270 and rotated in a width direction of the road by the driving motor 4270 to adjust an angle.

In other words, the sound sensor unit 4220 according to embodiment 1 of other embodiments receives sound of an initially set road surface area, and adjusts the angle and road width direction positions of the first and second transmitting and receiving members 4250 and 4260 by the controller 4300, thereby receiving sound of a road surface area other than the initially set road surface area, thus grasping a road surface state of a wider range than the related art, thereby improving reliability of grasping the road surface state.

In this case, the coupling relationship between the first rail 4122 and the coupling assembly 4210 is formed by threads of different directions on the outer circumference from one side to the other side of the first rail 4122, and one side of the first rail 4122 and the other side of the coupling assembly 4210 pass through and are screwed with each other, so that the coupling assembly 4210 moves in the lateral direction (i.e., to one side or the other side direction) along the first rail 4122 rotated by the rotation motor 4126, and the pair of coupling assemblies 4210 moves in opposite directions along the threads formed on the outer circumferences of both sides of the first rail 4122.

Further, in more detail, in embodiment 2 of other embodiments, the horizontal frame 4120 of the structure 4100 for mounting the sound sensor unit 4200 includes a first rail 4122 mounted in front of the lower portion in the longitudinal direction, a second rail 4124 mounted in rear of the lower portion in the longitudinal direction, and a rotation motor 4126 coupled to the first rail 4122 and the second rail 4124, respectively, the rotation motor 4126 being controlled by a controller 4300 described in detail later to rotate the first rail 4122 and the second rail 4124.

The sound sensor unit (4200) of embodiment 2 of other embodiments (i.e., the first transceiving member (4250) and the second transceiving member (4260)) includes: a coupling assembly 4210 mounted in either the first track 4122 or the second track 4122 such that it moves according to rotation of the first track 4122 or the second track 4124; a main body 4230 installed below the coupling assembly 4210; a drive motor 4270 controlled by the controller 4300; and a sound sensor 4240 installed below the driving motor 4270 and rotated in the width direction of the road by the driving motor 4270 to adjust an angle.

In other words, the sound sensor unit 4200 of embodiment 2 of other embodiments is configured to receive sound of a road surface area of a first set road and adjust the angle and the road width direction positions of the first transmitting-receiving member 4250 and the first transmitting-receiving member 4260 by the controller 4300 so as to receive sound of other road surface areas other than the road surface area of the first set road, thereby achieving an effect of improving reliability of grasping the road surface state by recognizing a road surface condition of a wider range than conventional.

In this case, the coupling relationship between the first and second rails 4122 and 4124 and the coupling assembly 4210 is formed by threads on the outer circumferences of the first and second rails 4122 and 4124, and the first and second rails 4122 and 4124 are screwed and coupled to the coupling assembly 4210 such that the coupling assembly 4210 moves in the lateral direction (i.e., to one side or the other side) along the first and second rails 4122 and 4124 rotated by the rotation motor 4126.

Further, the coupling relationship between the drive motor 4270 and the sound sensor 4240 of embodiment 1 and embodiment 2 of the other embodiments is such that the drive motor 4270 and the sound sensor 4240 rotated leftward and rightward according to the operation are coupled through the gear box 4280, and the drive motor 4270 is controlled by the controller 4300 described in detail later, thereby adjusting the installation angle of the sound sensor 4240.

Further, similar to the sound sensor unit 4200 of the previously described embodiment, the first and second transceiver members 4250 and 4260 of the other embodiments 1 and 2 may include a connection lever 4220 rotatably hinged to the coupling assembly 4210 in the front-rear left-right direction.

In more detail, the first and second transceiving members 4250 and 4260 of other embodiments 1 and 2 are configured to include a connection rod 4220, an upper portion of the connection rod 4220 is coupled to a lower portion of the coupling assembly 4210 so as to be rotatable in front-rear and left-right directions, and a main body 4230 of the first and second transceiving members 4250 and 4260 is coupled to a lower end of the connection rod 4220 so that the coupling assembly 4210, the connection rod 4220, and a main body of the sound sensor unit 4200 of one embodiment have the same structure. In this case, the coupling assembly 4210 and the connection rod 4220 are configured the same as the configuration of the sound sensor unit 4200 of the previous embodiment.

That is, when vibration is generated in the structure 4100 by the connection rod 4220, the sound sensor unit 4200 (i.e., the first transceiving member 4250 and the second transceiving member 4260 of embodiments 1 and 2 of other embodiments), the connection rod 4220, the main body, and the sound sensor 4240 are prevented from shaking, and thus radiation and reception of sound can be performed more stably.

Further, the main body 4230 of embodiments 1 and 2 of other embodiments includes an upper receiving groove 4234 formed in an upper portion and a ball bearing mounted in the upper receiving groove 4234, and the connection rod 4220 may include a lower ball 4226, which is inserted into and coupled to an inner side of the upper receiving groove 4234 of the main body 4230 at a lower end, so that the main body 4230 is coupled to the connection rod 4220 in the form of a ball hinge so as to be rotatable in front-rear and left-right directions to prevent secondary shaking, so that sound emission and reception from the sound sensor 4240 may be more stably performed.

Further, the connection rod 4220 of embodiments 1 and 2 of other embodiments includes a protection member 4222 coupled to an upper outer circumference and coupled to the outer circumference of the connection rod 4220 so as to pass through the center and be located at an upper portion of the sound sensor 4240 and formed in a circular shape of a curved surface protruding toward the upper portion so that the lower portion is opened and a reflection layer 4222a capable of reflecting sound is formed on an inner surface thereof.

That is, the protection member 4222 is coupled to the link 4220 through its center and is coupled to be positioned to the upper portion of the sound sensor 4240 such that even if a signal of sound reflected from the road surface is not directly received by the sound sensor 4240, the signal of sound can be reflected and received by the reflection layer 4222a, so that as many sound signals as possible can be received.

This can achieve the effect of receiving as much sound as possible that moves generally straight in the line of sight, and thus improve the reliability of measurement by sound, so that the reliability of understanding the road surface state can be improved.

Further, the protective member 4222 may protect an upper portion of the sound sensor 4240 so that the sound sensor 4240 may minimize damage caused by ultraviolet rays or rainwater or interference with the reception of sound by the excreta of birds.

In this case, since the protection member 4222 is included, the main body 4230 includes a plurality of auxiliary receiving sensors 4232 installed at an upper portion, and since the sound reflected by the reflection layer 4222 is difficult to be received by the main body 4230 located at an upper portion of the sound sensor 4240, the auxiliary receiving sensors 4232 can achieve an effect of receiving the reflected sound more stably.

Further, the rotating motors 4126 of embodiments 1 and 2 of the other embodiments have rotating rods connected to the first rail 422 and the second rail 4124, respectively, and the upper portions thereof are fixedly connected to the lower portions of the horizontal frames 4120.

Meanwhile, the sound sensor unit 4200 according to the embodiment different from the present disclosure is characterized in that, when sound is radiated to the road surface of the road, frequencies of the sound are differently radiated by the controller 4300 for each predetermined time, and this makes it possible to generate sound information capable of determining the road surface state in more detail by different frequencies, thereby achieving an effect of grasping a more reliable road surface state.

For example, according to the sound sensor unit 4200 of the embodiment different from the present disclosure, sound of 40kHz is emitted to the road surface at a normal time, sound of 80kHz is radiated to the road surface by the controller 4300 when the sound reaches a predetermined time, and sound of 120kHz is radiated to the road surface by the controller 4300 when the sound reaches a predetermined time, thus generating sound information capable of determining a detailed state of the road surface.

That is, the sound sensor unit 4200 according to the embodiment different from the present disclosure is characterized in that when sound is radiated to a road surface of a road, frequencies of the sound are differently radiated every predetermined time, and by radiating the sound a predetermined n times, the state of the road surface can be grasped in more detail, thereby achieving an effect of further improving reliability of grasping the state of the road surface.

The controller 4300, which is a main component of the road infrastructure sensor construction structure of the present disclosure, is installed in the vertical frame 4110 of the structure 4100, receives sound information from the sound sensor unit 4200, and transmits the sound information to the central management server, specifically, extracts characteristics of sound signals from the sound information received by the sound sensor 4240, classifies the sound information to grasp the state of the road surface, then estimates the state of the road surface, and transmits the estimated state of the road surface to the central management server, thereby taking measures to prevent accidents caused by the current state of the road surface.

The controller 4300 according to the present disclosure not only controls the above-described sound sensor 4240, but also automatically controls the rotation motor 4126 and the driving motor 4270 to predetermined input values or controls the rotation motor 4126 and the driving motor 4270 through input information of the central management server.

Further, the controller 4300 according to the present disclosure may include and control a signal converter that performs frequency conversion on a predetermined area on a domain of sound information received from the sound sensor 4240 to acquire a frequency domain signal, and an artificial neural network that extracts characteristics of the input signal based on a road surface classification model learned by using the frequency domain signal as the input signal and classifies the input signal to estimate a state of the road surface.

That is, the controller 4300 may be included in a switchboard or terminal box and mounted together in the vertical frame 4110 of the structure 4100, and may include a wired or wireless communication unit to transmit sound information to the central management server.

The construction method of the road infrastructure sensor construction structure of the present disclosure will be described in detail below.

The construction method of the road infrastructure sensor construction structure of the present disclosure is constructed to include a structure mounting step 4S10 of mounting the structure 4100, a sensor mounting step 4S20 of mounting the sound sensor unit 4200 on the structure 4100, and a terminal box mounting step 4S30 of mounting the controller 4300 on the structure 4100.

The structure mounting step 4S10 mounts the vertical frame 4110 on a road or a road edge, and mounts the structure 4100 including the vertical frame 4110 and the horizontal frame 4120 by coupling the horizontal frame 4120 to an upper portion of the vertical frame 4110 in a width direction of the road.

In this case, if the shape of the structure 4100 is the same as that of the streetlamp shown in fig. 41 to 44The same shape is installed by the above method if the shape of the structure 4100 is the same as the advertising board structure shown in fig. 45 to 48The same shape, a pair of vertical frames 4110 are installed, and thus a horizontal frame 4120 connecting the upper portions of the pair of vertical frames 4110 is installed.

Further, it is apparent that when the sound sensor unit 4200 of the other embodiments described above is mounted, the first rail 4122 and the second rail 4124 are mounted below the horizontal frame 4120.

Meanwhile, since the structure 4100 such as a street lamp or a photovoltaic panel structure has been previously installed, the structure installation step 4910 of the present disclosure may be omitted.

The sensor mounting step 4920 includes mounting the sound sensor unit 4200 that receives reflected sound after radiating sound to the road surface to the horizontal frame 4120 of the structure 4100 after the structure mounting step 4910, and mounting the coupling assembly 4210 under the above-described horizontal frame 4122.

In this case, when the sound sensor unit 4200 of the present embodiment is mounted, the coupling assembly 4210 is fixedly coupled to the lower portion of the horizontal frame 4120, and when the sound sensor unit 4200 of other embodiments is mounted, the coupling assembly 4210 is movably mounted in the first and second rails 4122 and 4124, which are mounted under the horizontal frame 4120.

The coupling assembly 4210 includes a connection rod 4220 coupled to a lower portion when the sound sensor unit 4200 of the present embodiment is installed, a protection member 4222 installed on an outer circumference of the connection rod 4220, a main body 4230 coupled to a lower portion of the connection rod 4220, a plurality of auxiliary receiving sensors 4232 coupled to an upper portion of the main body 4230, and a sound sensor 4240 coupled to a lower portion of the main body 4230. Or in the case of mounting the sound sensor unit 4200 according to other embodiments, the connection rod 4220 coupled to the lower portion, the protection member 4222 mounted on the outer circumference of the connection rod 4220, the main body 4230 coupled to the lower portion of the connection rod 4220, the plurality of auxiliary receiving sensors 4232 coupled to the upper portion of the main body 4230, the driving motor 4270 coupled to the lower portion of the main body 4230, and the sound sensor 4240 coupled to the lower portion of the driving motor 4270 have been coupled.

The terminal box mounting step 4930 is to mount a controller 4300 after the sensor mounting step 4920, the controller 4300 and the sound sensor unit 4200 are connected to the vertical frame 4110 of the structure 4100, and connected to a central management server in a wired or wireless manner, and the sound sensor unit 4200 is electrically connected so that the sound sensor unit 4200 can be controlled by the controller 4300.

In this case, when the sound sensor unit 4200 of one embodiment is mounted, the terminal box mounting step 4930 may further include an initial emission setting step of setting sound emission times such that arrival times of sounds reflected and received by the plurality of sound sensor units 4200 are all the same, or when the sound sensor unit 4200 of other embodiments is mounted, the terminal box mounting step 4930 includes an initial emission setting step of setting the first transceiving member 4250 and the second transceiving member 4260 of the initial mounting to receive sounds emitted from each other.

Accordingly, the road infrastructure sensor system construction structure of the present disclosure and the construction method thereof can smoothly recognize the road surface state in a noncontact manner by sound, and can improve the reliability of estimating the road surface state by recognizing the road surface state in a wider measurement range than conventional, and can improve the measurement reliability of the road surface by receiving a large amount of sound by the protective member 4222 having the reflective layer 4222a, even though the reflective layer 4222a is difficult to receive small disturbance due to the characteristic of the sound having linearity, and can obtain the effect of improving the measurement reliability by minimizing the natural frequency generated by the vibration of the structure 4100 by minimizing the reception of the natural frequency by the protective member 4221 when the structure 4100 vibrates due to the passing or disturbance of the vehicle.

In the above-described specific embodiments of the present disclosure, the components included in the present disclosure are expressed in single or in plural according to the presented specific embodiments. However, for ease of explanation, single or multiple expressions are selected to accommodate the presented situation, and the disclosure is not limited to single or multiple expressions, and components in multiple expressions may be composed of single or multiple.

Meanwhile, the embodiments of the present disclosure disclosed in the specification and the drawings are presented only to easily explain the technical contents of the present disclosure and to help understand the present disclosure, and are not intended to limit the scope of the present disclosure. That is, it is apparent to those skilled in the art that other modifications based on the technical spirit of the present disclosure may be practiced. Furthermore, the embodiments disclosed in the present specification may be combined and operated as necessary. For example, portions of one embodiment that are different from one embodiment of the present disclosure may be combined and implemented in the form of an embodiment that is not specified in the present specification.

Meanwhile, the order of description of the methods of the present disclosure in the drawings does not necessarily correspond to the order of execution, and the relationship between the front and rear or the parallel execution may be changed.

Alternatively, the drawings illustrating the methods of the present disclosure may omit some components and include only some components within a range that does not impair the spirit of the present disclosure.

Further, the method of the present disclosure may be implemented by combining some or all of the contents included in each embodiment within a range that does not impair the spirit of the present disclosure.

Claims

1. An electronic device for classifying a road surface using acoustic signals, the electronic device comprising:

A transceiver configured to transmit and receive sound signals;

an atmospheric sensor; and

At least one processor electrically connected to the transceiver and the atmospheric sensor, wherein the at least one processor is configured to:

Transmitting, using the transceiver, an acoustic signal to a target pavement spaced a first distance from the electronic device;

Receiving a reflected signal of the sound signal of the target road surface using the transceiver;

acquiring atmospheric information related to the sound signal using the atmospheric sensor;

Acquiring first data of the received reflected signal;

generating second data by correcting the first data based on the atmospheric information;

acquiring third data related to frequency domain information of the second data based on the second data; and

Determining a type of the target road surface based on the third data and a road surface classification artificial neural network; and

Wherein the road surface classification artificial neural network is trained to a frequency domain data set generated based on sound signals reflected from a road surface at a second distance different from the first distance.

2. The electronic device of claim 1, wherein the second data is generated by correcting the first data based on the atmospheric information and the first distance.

3. The electronic device of claim 1, wherein the first distance is estimated based on a time of transmission of the sound signal and a time of receipt of the reflected signal.

4. The electronic device of claim 1, wherein the third data is obtained by converting the second data to STFT (short time fourier transform).

5. The electronic device of claim 1, wherein the at least one processor is configured to generate a signal for controlling a road surface management device mounted on the target road surface based on the determined type of the target road surface, and

The pavement management device comprises a heating wire device or a salt water spraying device.

6. The electronic device of claim 5, wherein the at least one processor is configured to:

Determining whether a preset weather condition is met;

Generating a signal for controlling the road surface management device when the preset weather condition is satisfied;

Determining whether the type of the target road surface determined at the first time is changed at the second time; and

When the first category determined as the type of the target road surface at the first time is different from the second category determined as the type of the target road surface at the second time, it is determined whether to generate a signal for controlling the device mounted on the target road surface based on the type of the target road surface determined at the third time.

7. The electronic device of claim 1, wherein the type of the target road surface is determined at each first cycle, and the at least one processor is configured to determine the type of the target road surface at each second cycle when the type of the target road surface is determined to be the first category.

8. The electronic device of claim 1, wherein the electronic device further comprises:

At least one of an infrared sensor for acquiring temperature information of the target road surface or a visual sensor for acquiring image information of the target road surface, and

The at least one processor is configured to determine a type of the target pavement further based on the temperature information or the image information.

9. A method performed by an electronic device for classifying a road surface using acoustic signals, the method comprising:

transmitting an acoustic signal to a target road surface spaced a first distance from the electronic device;

Receiving a reflected signal of the sound signal of the target road surface;

Acquiring atmospheric information related to the sound signal;

Acquiring first data of the received reflected signal;

Determining a type of the target road surface based on the third data and a road surface classification artificial neural network, and

10. The method of claim 9, wherein generating the second data further comprises correcting the first data corrected based on the atmospheric information based on the first distance.

11. The method of claim 9, further comprising estimating the first distance based on a time of transmission of the sound signal and a time of receipt of the reflected signal.

12. The method of claim 9, wherein the third data is obtained by converting the second data to STFT (short time fourier transform).

13. The method of claim 9, further comprising:

generating a signal for controlling a road surface management device mounted on the target road surface based on the determined type of the target road surface, and

Wherein the pavement management device comprises a heating wire device or a salt water spraying device.

14. The method according to claim 13,

Wherein generating the signal for controlling the road surface management device comprises:

Determining whether a preset weather condition is met;

When the preset weather conditions are met, determining whether the type of the target pavement determined at the first time is changed at the second time; and

15. The method of claim 9, wherein determining the type of the target pavement comprises:

Determining a type of the target road surface based further on the temperature information of the target road surface or the image information of the target road surface,

Wherein the type of the target road surface is determined at each first period, and

When the type of the target road surface is determined as the first category, the type of the target road surface is determined at each second period.