TWI785761B - Vehicle intelligent two steps security control system - Google Patents

Abstract

A vehicle intelligent control system includes an electronic device, a pair of image capturing device, a pair of image receiving and transmitting device, a cloud platform and a server. The pair of image receiving and transmitting device are electrically connected to the pair of image capturing device. The server receives and transmits the image flow, and the pair of image receiving and transmitting device includes a processing unit, a memory and a transceiver unit to transmit the image flow by TCP. The server includes a processing unit, a memory and a transceiver unit. The memory stores a convolution neural network. When the driver takes the electronic device and is near the door of the car, the transceiver unit makes the communication with the electronic device by Bluetooth. The transceiver unit transmits the token to the electronic device. The electronic device makes an authentication process on the token by the cloud platform. When the server receives the authenticated token, the processing unit controls the pair of image capturing device to shoot the environment image. The processing unit utilizes the convolution neural network to calculate a similarity score based on the driver face in the environment image. The processing unit controls the door to open or close based on the similarity score.

Description

Two-stage safety control system for vehicle intelligence

The present invention relates to a vehicle control system, in particular to a two-stage intelligent security control system for a vehicle that utilizes cloud key exchange authentication and convolutional neural network to identify the driver's face for two-stage authentication to control the opening or closing of the vehicle door.

In the era of rapid technological change, the car has become an indispensable means of transportation. At present, the car still needs to use the metal key or chip key to open the door and start the engine. However, when the metal key or chip key is missing or not carried, the driver needs to Find another locksmith to unlock, thereby causing inconvenience to the driver and waste of money.

In view of the foregoing, the inventor of the present invention conceived and designed a two-stage safety control system for vehicle intelligence, in order to improve the deficiencies of conventional technologies, and further enhance the implementation and utilization in the industry.

Based on the above purpose, the present invention provides a two-stage security control system for vehicle intelligence, which uses cloud key exchange authentication and convolutional neural network to identify the driver's face for two-stage authentication, controls the opening or closing of the door, and does not require the driver to carry metal Key or chip key, to solve the problems faced in the prior art.

Based on the above purpose, the present invention provides a two-stage intelligent vehicle safety control system, which includes an electronic device, a pair of image capture devices, a pair of image receiving and transmitting processors, a cloud platform, and a servo processing calculator. A pair of image capture devices are respectively installed on the rearview mirror of the car. A pair of image receiving and transmitting processors are electrically connected to a pair of image capturing devices. A pair of image receiving and transmitting processors includes a processing unit, a memory, and a transceiver unit To implement the video streaming of the Wireless Transmission Control Protocol. The servo processing arithmetic unit is embedded in the car and electrically connected to a pair of image receiving and transmitting processors. The servo processing arithmetic unit includes a processing unit, a memory, and a transceiver unit. The memory stores a convolutional neural network. When the driver carries an electronic device When approaching the door of the car, the transceiver unit sends out a wireless signal, the electronic device receives the wireless signal and completes a pairing operation with the servo processing unit, and the transceiver unit sends the key to the electronic device. The cloud platform network connects the servo processing unit and the electronic device, the electronic device sends the key to the cloud platform for authentication, and the cloud platform sends the key to the server processing unit after passing the verification process. When the servo processing unit receives the key after passing the authentication procedure, the processing unit controls the vehicle’s rearview mirror, a pair of image capture devices, and a pair of image receiving and transmitting processors to turn on, and a pair of image capture devices take pictures of the driver The surrounding environmental image, a pair of image receiving and transmitting processors convert the environmental image into an image stream and transmit it to the servo processing unit through the wireless transmission control protocol, and the processing unit of the servo processing unit uses the convolutional neural network ratio The similarity is calculated for the driver's face in the environment image and the driver's reference image stored in the memory, and the processing unit executes the control action on the car door according to the similarity.

Optionally, if the processing unit judges that the similarity is greater than a threshold value, the control action is to open the door.

Optionally, if the processing unit judges that the similarity is smaller than the threshold value, the control action is to close the vehicle door.

Optionally, the servo processing arithmetic unit further includes a warning unit, and when the servo processing arithmetic unit receives the key that fails the authentication procedure, the processing unit controls the warning unit to start.

Optionally, when the driver carries the electronic device away from the door, the transceiver unit does not receive the key, and the processing unit controls the door to close.

Optionally, the memory further stores an object recognition algorithm and a face recognition algorithm, and the processing unit uses the object recognition algorithm and the face recognition algorithm to identify multiple individuals and multiple objects in the environment image.

Optionally, the servo processing unit further includes a first database and a second database, the cloud platform includes a cloud database, the first database stores a plurality of driver characteristics, and the second database stores a plurality of reference object characteristics, The cloud database stores multiple reference face features, the processing unit uses the preliminary convolutional neural network to initially identify multiple objects and multiple individuals in the environmental image, and the processing unit uses the object recognition algorithm to compare the multiple objects and multiple individuals in the environmental image Refer to the object features to identify multiple objects. The processing unit uses the convolutional neural network and the face recognition algorithm to compare the reference features of multiple individuals and multiple drivers in the environmental image to calculate multiple similarities. The processing unit is based on A plurality of similarities selects one of the plurality of individuals as the face of the driver.

Optionally, after the processing unit identifies the plurality of objects and the plurality of individuals, the processing unit calculates the first distances between the plurality of individuals and the vehicle and the second distances between the plurality of objects and the vehicle.

Optionally, the processing unit judges that the plurality of objects and the plurality of persons fall in the visual zone or the visual blind zone according to the plurality of first distances and the plurality of second distances.

Optionally, the servo processor further includes a single multi-frame detection unit, which uses a plurality of bounding boxes to frame a plurality of persons and a plurality of objects in the environment image respectively.

Based on the above, the intelligent two-stage safety control system for vehicles of the present invention controls the opening or closing of the car door through the two-stage authentication of cloud key exchange authentication and convolutional neural network recognition of the driver's face, without the need for the driver to carry metal key or wafer key.

The advantages, features and technical methods achieved by the present invention will be described in more detail with reference to exemplary embodiments and accompanying drawings to make it easier to understand, and the present invention can be implemented in different forms, so it should not be understood as being limited to what is shown here The stated embodiments, on the contrary, for those skilled in the art, the provided embodiments will make the present disclosure more thorough and comprehensive and completely convey the scope of the present invention, and the present invention will be only the appended The scope of the patent application is defined.

It should be understood that although the terms "first", "second" and the like may be used in the present invention to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections Should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer and/or section from another element, component, region, layer and/or section. Accordingly, "first element", "first component", "first region", "first layer" and/or "first portion" discussed below may be referred to as "second element", "second component" , "second region", "second layer" and/or "second part", without departing from the spirit and teachings of the present invention.

In addition, the terms "comprising" and/or "comprising" refer to the presence of stated features, regions, integers, steps, operations, elements and/or parts, but do not exclude one or more other features, regions, integers, steps, operations , the presence or addition of elements, parts and/or combinations thereof.

Unless otherwise defined, all terms (including technical and scientific terms) used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted to have definitions consistent with their meanings in the context of the relevant art and the present invention, and will not be interpreted as idealistic or overly formal unless otherwise expressly defined herein.

Please refer to Figure 1A and Figure 1B, which are block diagrams of the two-stage safety control system for vehicle intelligence of the present invention and a schematic diagram of the use of the two-stage safety control system for vehicle intelligence of the present invention. As shown in Figure 1A and Figure 1B, the two-stage safety control system for vehicle intelligence of the present invention includes an electronic device 10, a pair of image capture devices 30, a pair of image receiving and transmitting processors 31, and a cloud platform 40 and the servo processing arithmetic unit 20. A pair of image capturing devices 30 are installed on two rearview mirrors of the car C respectively. A pair of image receiving and transmitting processors 31 are electrically connected to a pair of image capture devices 30. A pair of image receiving and transmitting processors 31 includes a processing unit 311, a memory 312, and a transceiver unit 313 to execute the image string of the wireless transmission control protocol. Flow IS. The servo processing arithmetic unit 20 is embedded in the car C and is electrically connected to a pair of image receiving and transmitting processors 31. The servo processing arithmetic unit 20 includes a processing unit 21, a memory 22, and a transceiver unit 23. The memory 22 stores a convolutional neural network. Road CN1, when the driver D carries the electronic device 10 close to the door of the car C (that is, the position of the driver D falls into the sensing area SR of the transceiver unit 23), the transceiver unit 23 sends a wireless signal WLS, and the electronic device 10 receives The wireless signal WLS completes the pairing operation with the servo processor 20 , and the transceiver unit 23 transmits the key T to the electronic device 10 . The cloud platform 40 connects the servo processing unit 20 and the electronic device 10 through the network, the electronic device 10 transmits the key T to the cloud platform 40 for authentication procedure AS, and the cloud platform 40 transmits the key T to the servo processing unit 20 after passing the authentication procedure AS . When the servo processing unit 20 receives the key T after passing the authentication procedure AS, the processing unit 21 turns on the two rearview mirrors of the car C and controls a pair of image capture devices 30 and a pair of image receiving and transmitting processors 31 to turn on, A pair of image capture devices 30 captures the environment image EI around the driver D, and a pair of image receiving and transmitting processors 31 convert the environment image EI into an image stream IS and transmit the It is transmitted to the servo processing unit 20, and the processing unit 21 uses the convolutional neural network CN1 to compare the driver’s face in the environment image EI with the driver’s reference image stored in the memory 22 to calculate the similarity SS, and the processing unit 21 calculates the similarity SS according to the similarity SS executes the control action on the car door.

Wherein, the electronic device 10 may be a mobile phone or a tablet computer, or other preferred types of electronic devices, and is not limited to the scope of the present invention. The wireless signal WLS may include a Bluetooth signal, a Wifi signal, a ZigBee signal or a LoRa (Long Range) signal, and of course it may also be other types of wireless signals, and is not limited to the scope of the present invention.

It is worth mentioning that the processing unit 21 includes a micro control unit (micro control unit, MCU) and a micro processing unit (micro processing unit, MPU). The serial port of the control unit is a full-duplex asynchronous serial port communication method. The microprocessor unit adopts NVIDIA's Jetson Nano, which is a multi-purpose microprocessor and has a quad-core ARM Cortex-A53 (ARMv8) 64-bit processor, which can handle the signal of the microcontroller unit, the data access of the user end and the server end, Web page data visualization calculation and processing and network connection, due to Jetson Nano's asynchronous multi-processing and multi-execution continuation, will be executed in different threads in the background, so the application can call the thread of the asynchronous method to continue execution, and can Establish connections with multiple clients at the same time.

The wireless transmission control protocol used by the server processor 20 is the configuration of the client/server environment of the ImageZMQ TCP Streams Service and the configuration of request/response between the client and the server. In the micro-processing unit cluster architecture, the four cores of the two MPU Worker Nodes execute asynchronous procedures (Asynchronous Procedure) in parallel (concurrent) to perform operations to speed up real-time and massive data processing. In order to cooperate with the processing program of the working node of the micro-processing unit and because the time course of reading/writing data during TCP client/server signal transmission is executed with blocking I/O (blocking I/O), that is, when the server When listening to a port, the entire program will stop here completely, waiting for the client to send data in, so when the execution thread executing the synchronous (Synchronous) request needs to wait for the completion of the network operation, it can no longer perform other tasks (such as the user interface). Execution thread, etc.), the application program arranged on the execution thread will suspend and stop responding to user input. In addition, when other clients want to connect to the server, because the only network socket (WebSocket) is closed If it is occupied, other clients will not be able to connect.

Because in the method of asynchronous connection, the program will not be blocked because of the execution thread that needs to wait for the completion of the synchronous operation. This is because the asynchronous operation (Asynchronous Operation) will be executed in a different thread in the background, so the application can continue to execute on the thread that called the asynchronous method. The advantage of this is that it is efficient and the server can establish connections with multiple clients at the same time. This system uses ASYNCIO's Asynchronous TCP Streams (Asynchronous TCP Streams) technology to complete the synchronous Remote Procedure Call (RPC) operation of the remote network connection. In addition to the above-mentioned properties, the biggest advantage of this streaming technology is that it does not need to use low-level communication protocols to execute and complete data transmission and reception, so that the application can maintain fast data response and can improve scalability and reliability and achieve simultaneous multi- connection purpose. The programs executed by the server-side and client-side TCP: the TCP server-side program of the micro-processing unit cluster is installed in the asynchronous multiprocessing (multiprocessing) and multi-threading (multithreading) time course of the master node for real-time deployment , Respond to or terminate the operation of the client program installed on the two worker nodes (Worker Node).

In this embodiment, the servo processor 20 further includes a warning unit 24, a first database 25, a second database 26, and a single multi-frame detection unit 27. The memory 22 also stores object recognition algorithms AG1, people The face recognition algorithm AG2, the convolutional neural network CN2, the preliminary convolutional neural network SCN1 and the preliminary convolutional neural network SCN2, the cloud platform includes 40 including the cloud database 41 . The warning unit 24 includes a red light and a green light. The warning unit 24 is arranged on the vehicle door to indicate whether the vehicle door is opened or the authentication of the key T is successful or not. The first database 25 stores a plurality of driver reference features, the second database 26 stores a plurality of reference object features RF, the cloud database 41 stores a plurality of reference face features RF, and the cloud database 41 also stores the second database 26 Other reference object features that do not exist. The single multi-frame detection unit 27 utilizes a plurality of bounding boxes to respectively frame a plurality of persons and a plurality of objects in the environment image EI.

Referring back to Fig. 1A and Fig. 1B, the procedures for the processing unit 21 to control the opening and closing of the car door are as follows: (1) first-stage authentication: when the driver D carries the electronic device 10 near the car door of the car C (i.e. The location of the driver D falls into the sensing area SR of the transceiver unit 23), the transceiver unit 23 sends out the wireless signal WLS, the electronic device 10 receives the wireless signal WLS and completes the pairing action with the servo processing operation gas 20, and the transceiver unit 23 transmits the gold The key T is sent to the electronic device 10, and the electronic device 10 sends the key T to the cloud platform 40 to perform the authentication procedure AS. If the cloud platform 40 finds the key T in its memory and completes the authentication procedure AS, the cloud platform 40 sends the completed authentication procedure AS After the key T is sent to the servo processing unit 20, the processing unit 21 controls the green light of the warning unit 24 to light up, if the cloud platform 40 does not find the key T in its memory and transmits the key T without completing the authentication procedure AS To the servo processor 20, the processing unit 21 controls the red light of the warning unit 24 to light up. (2) The second stage of authentication: the servo processing unit 20 receives the key after completing the authentication procedure AS, and the processing unit 21 opens the two rearview mirrors of the car C and controls a pair of image capture devices 30 and a pair of image receiving and transmitting The processor 31 is turned on, and a pair of image capture devices 30 shoots the environment image EI around the driver D, and a pair of image receiving and transmitting processors 31 convert the environment image EI into an image stream IS and transmit it through the wireless transmission control protocol (Transmission Control Protocol). Protocol, TCP) and transmit it to the servo processor 20, the processing unit 21 uses the convolutional neural network CN1 to compare the driver's face in the environment image EI and the driver's reference image stored in the memory to calculate the similarity SS, when the processing unit 21. Judging that the similarity SS is greater than the threshold value TH (for example, 0.6), the processing unit 21 identifies and confirms the identity of the driver D, and the processing unit 21 performs a control action on the door to open the door. When the processing unit 21 judges that the similarity SS is less than Threshold value TH (for example, 0.6), the processing unit 21 fails to recognize and confirm the identity of the driver D, and the control action performed by the processing unit 21 on the door is to close the door.

It should be noted that when the driver D carries the electronic device 10 away from the door (that is, the location of the driver D is not in the sensing area SR of the transceiver unit 23), the electronic device 10 does not receive the wireless signal WLS and does not perform calculations with the servo processing. The device 20 completes the pairing operation, that is, the transceiver unit 23 is not connected with the electronic device 10, and the processing unit 21 controls the door to close.

It is worth mentioning that, in order to improve the accuracy of identifying and confirming the driver D by the processing unit 21, the first database 25 also stores reference pictures of the driver D from different angles, that is, taking the front of the driver D as a reference, The four directions of left, right, up and down are turned 15 degrees and 30 degrees respectively. When the head of the driver D is turned, the processing unit 21 can still identify and confirm the identity of the driver D.

Please refer to FIG. 2, which is a flowchart of the cloud key exchange authentication of the present invention. As shown in Fig. 2, and with Fig. 1A and Fig. 1B, the wireless signal WLS is set as a Bluetooth signal, and the flow of cloud key exchange authentication is as follows: Step S11: If the servo processing unit 20 sends out a Bluetooth signal through the transceiver unit 23 signal, the electronic device 10 is connected to the transceiver unit 23 (that is, the servo processing unit 20 and the electronic device 10 complete the pairing action), the processing unit 21 controls the green light of the warning unit 24 to turn on, and enters step S13; if the electronic device 10 is not connected Go to the transceiver unit and go to step S12.

Step S12: the processing unit 21 controls the red light of the warning unit 24 to light up

Step S13: the processing unit 21 continues to generate the key T and register the key T on the cloud platform 40, the cloud platform 40 stores the key T, and the servo processor 20 transmits the key T to the electronic device 10 through the transceiver unit 23, and enters step S14 .

Step S14: the electronic device 10 transmits the key T and verification key requirements to the cloud platform 40, and the cloud platform 40 finds the key T in its memory, and if the cloud platform 40 finds the key T in its memory, go to S16 Step; if the cloud platform 40 does not find the key T in its memory, go to step S15.

Step S15: the cloud platform 40 transmits the key T after the unfinished authentication procedure AS to the servo processor 20, and the processing unit 21 controls the red light of the warning unit 24 to light up

Step S16: the cloud platform 40 transmits the post-authentication process AS key T (i.e., post-authentication key T) to the electronic device 10, and the electronic device 10 forwards the post-authentication process AS post-transmission key T to the servo processor 20, and proceeds to S17. step.

Step S17: the transceiver unit 23 receives the key T after the authentication procedure AS is completed, the processing unit 21 controls the green light of the warning unit 24 to light up, the processing unit 21 confirms whether the key T is valid after the authentication procedure AS is completed, and the processing unit 21 confirms that the authentication procedure is completed If the key T is valid after the AS, go to step S19; if the processing unit 21 confirms that the key T is invalid after the authentication procedure AS is completed, go to step S18.

Step S18: the processing unit 21 controls the red light of the warning unit 24 to light up

Step S19: the processing unit 21 turns on the two rearview mirrors of the car C and controls a pair of image capturing devices 30 and a pair of image receiving and transmitting processors 31 to turn on, and a pair of image capturing devices 30 take pictures and control the warning unit 24 The green light is on.

Please refer to FIG. 3 and FIG. 4, which are configuration diagrams of the algorithm of the present invention and flow charts of object recognition and face recognition of the present invention. As shown in FIG. 3, together with FIG. 1A and FIG. 1B, after the electronic device 10 and the cloud platform 40 complete the authentication procedure AS, the processing unit 21 of the servo processing unit 20 makes the two rearview mirrors of the car C open and Control a pair of image capture devices 30 and a pair of image receiving and transmitting processors 31 to start, and the processing unit 21 of the servo processing unit 20 uses the preliminary convolutional neural network SCN1 and SCN2 together with the complex numbers stored in the cloud database 41 A reference face feature RF and other reference object features perform preliminary recognition of multiple faces and objects in the environment image EI (extensive face/object recognition), and the processing unit 21 separates multiple faces and multiple objects for further recognition.

In terms of object recognition, the processing unit 21 uses the object recognition algorithm AG1 and the convolutional neural network CN2 to compare a plurality of objects in the environment image EI with a plurality of reference object features RO in the second database 26 (as shown in FIG. 4 ). shown) or other reference object features of the cloud database 41 to identify a plurality of objects, such as basketballs, trees or houses. In terms of face recognition, the processing unit 21 uses the face recognition algorithm AG2 to compare the plurality of individuals in the environmental image EI with the plurality of reference face features RF in the cloud database 41 to identify the plurality of individuals, and the processing unit 21 uses the convolutional neural network CN1 compares the plurality of individuals in the environmental image EI and the plurality of driver reference features DF stored in the first database 25 (as shown in FIG. 4 ) to calculate the plurality of similarities SS, and the processing unit 21 calculates the plurality of similarities SS according to the plurality of similarities. The SS selects one of the plurality of individuals as the driver's face, and the processing unit 21 controls the car door according to the similarity SS corresponding to the driver's face.

As shown in FIG. 4, together with FIG. 1A, FIG. 1B and FIG. 3, a pair of image capture devices 30 captures the environment image EI around the driver D, and the processing unit 21 transmits a two-dimensional code (QR code) to The electronic device 10, if the electronic device 10 successfully scans the two-dimensional code, the cloud database 41 and the first database 25 respectively provide the reference face feature RF and a plurality of driver reference features DF to the processing unit 21, and the processing unit 21 directly recognizes the Whether the driver D of the car C is the real driver, the processing unit 21 identifies the driver D who is close to the car C as the real driver, and the processing unit 21 calculates the first distance between the driver D and the car C, and the processing unit 21 passes through The auxiliary voice message informs the driver D that the authentication is successful, and the processing unit 21 controls the door to open.

If the electronic device 10 fails to scan the QR code, a pair of image receiving and transmitting processors 31 converts the environmental image EI into an image stream IS and transmits it to the server processing operation through the wireless transmission control protocol (Transmission Control Protocol, TCP) device 20, since the environment image EI photographed by the image capture device 30 is a dynamic image, the processing unit 21 saves the environment image EI through the technology of tensor flow freezing inference graphics, and the single multi-frame detection unit 27 (including Microsoft common object detection The data set) is combined with preliminary convolutional neural networks SCN1 and SCN2 and object recognition algorithm AG1 to preliminarily frame a plurality of individuals and a plurality of objects in the environmental image EI with a plurality of bounding boxes.

Next, regarding the part of identifying the plurality of objects, the processing unit 21 again uses the object recognition algorithm AG1 and the convolutional neural network CN2 to compare the plurality of objects in the environment image EI with the plurality of reference object features RO of the second database 26 To identify a plurality of objects, if some objects are not recognized by the processing unit 21, the processing unit 21 uses the characteristics of the cloud object stored in the cloud database 41 and the object recognition algorithm AG1 and the convolutional neural network CN2 to compare the unrecognized objects out objects. Regarding the part of identifying multiple individuals, the processing unit 21 uses a face recognition algorithm to detect the position, number, size and facial features of human faces to identify multiple individuals, and the processing unit 21 uses the convolutional neural network CN1 to compare the multiple individuals and The plurality of drivers in the first database 25 refer to the feature DF to calculate the similarity SS, and the processing unit 21 selects one of the plurality of individuals as the driver’s face according to the similarity SS, and the processing unit 21 controls the car door according to the similarity SS corresponding to the driver’s face , the processing unit 21 uses a plurality of reference face features RF (as shown in FIG. 3 ) stored in the cloud database 41 to compare and identify people who are not identified as drivers in the environmental image EI, and the processing unit 21 calculates When the processing unit 21 judges that the first distance between the plurality of individuals and the car C and the second distance between the plurality of objects and the car C is greater than the threshold value, the processing unit 21 notifies through an auxiliary voice message The driver D is authenticated successfully, and the processing unit 21 controls the door to open.

Next, the processing unit 21 judges that a plurality of objects and a plurality of individuals fall in the visible area or the visual blind area according to the plurality of first distances and the plurality of second distances, and the processing unit 21 judges the object according to the distance between the object and the car C and the distance between the person and the car Whether and whether the person is in the dead corner of driver D, assist driver D in driving.

Please refer to FIG. 5 and FIG. 6, which are configuration diagrams of the convolutional neural network of the present invention and schematic diagrams of the operation of the convolutional neural network of the present invention. As shown in FIG. 5, together with those shown in FIG. 1A to FIG. 4, the processing unit 21 first selects a partial image FI with multiple individuals from the environmental image EI, and the processing unit 21 divides the partial image as shown in FIG. 6 It is a plurality of small images P1, and each small image P1 obtains an array A through the 5-layer convolutional layer conv, the 3-layer maximum pooling layer MPL, and the 2-layer fully connected layer FL included in the convolutional neural network CN1 .

As shown in Figure 6, together with Figure 1A to Figure 4, a plurality of small images P1 are then passed through the convolutional neural network CN1 to obtain a plurality of arrays A, and each array A is selected by the maximum pooling layer MPL The most similar part of the plurality of driver features in the first database 25, reducing the dimensions of the plurality of arrays A, recombining the most similar part of each array A into a complete image, and the processing unit 21 compares the complete image and a plurality of driver features to confirm the identity of the driver, the processing unit 21 compares the similarity SS between the complete image and the plurality of driver features greater than the threshold value TH1, the processing unit 21 confirms that the driver in the complete image is a driver, and processes The unit 21 compares the similarity SS between the complete image and the plurality of driver features to be lower than the threshold TH1, and the processing unit 21 denies the human driver in the complete image.

Please refer to FIG. 7, which is a schematic diagram of the blind spot detection of the present invention. As shown in FIG. 7, together with FIG. 1 to FIG. 4, the driver D is in the state of driving the car C, a pair of image capture devices 30 capture the driving image, and the processing unit 21 uses the convolutional neural network CN2 to obtain The driving image recognizes the truck V, the processing unit 21 calculates the distance between the truck V and the car C as the second distance, and the processing unit 21 judges that the second distance falls in the blind spot R1, and the processing unit 21 sends an auxiliary voice message to the driver to remind the driver Attention should be paid to the truck V.

In addition, the processing unit 21 can use the dynamic object recognition algorithm to calculate the second distance between the truck V and the vehicle C in real time. The second distance is updated with the movement of the vehicle C and the truck V, so that the driver D is in the blind spot immediately. Detect dynamic objects (eg trucks). Among them, the dynamic object recognition algorithm provides the processing unit 21 to adjust the detection parameters in the visual blind zone R1 according to the position of the truck V detected by the processing unit 21 in the visual blind zone R1 and the field of vision of the driver D, and cooperates with the motion recovery structure ( Structure-from-Motion, SfM) principle to effectively measure the distance between a pair of image capture devices 30 and the rear object (such as a truck V) to determine the position of the dynamic object.

。 Please refer to FIG. 8, which is a schematic diagram of the measurement of the motion recovery structure of the present invention. As shown in Fig. 8, together with Fig. 1A, Fig. 1B and Fig. 7, a pair of image capturing devices 30 are arranged on the left and right of the car C, and a pair of image capturing devices 30 are located on the same The coordinates of the target object (such as the truck V) relative to the pair of image capture devices 30 can be found by taking photos on the horizontal line and at almost the same time point, and according to the triangle similarity principle. As an example, assuming that a pair of image capture devices 30 are placed in parallel on the xz plane after being oriented, the distance between the two image capture devices 30 is b, and the focal lengths of the two image capture devices 30 are both f. The coordinates of the image capture device 30 are p1 (x1, y1) and p2 (x2, y2), and the coordinates of the truck V are P (x, y, z), the following relationship can be obtained:

.

。由於兩個影像擷取裝置30的中心點𝑂1、𝑂2 與目標點P可形成三維空間中的一個平面𝑂1𝑂2P，極平面(Epipolar Plane)，此時極平面與二幀畫面相交於二條直線稱為極線。從移動中的兩個影像擷取裝置30連續拍攝的一幀畫面中，假設目標點p在兩個影像擷取裝置30中的成像點分別是p1與p2，給定p1的座標值，並且採用基於運動恢復結構原理，亦即距離越遠物體視差越小，反之視差則越大，我們即可量測到極幾何原理中的極線約束參數(Epipolar Constraint)，約束參數就是匹配點p2在三維空間中循著極線移動的方向。 Setting the parallax of the two image capture devices 30 as d=x1-x2, the following relationship can be obtained:

. Since the center points 𝑂1, 𝑂2 of the two image capture devices 30 and the target point P can form a plane 𝑂1𝑂2P in the three-dimensional space, the epipolar plane (Epipolar Plane), at this time the epipolar plane intersects the two frames of pictures on two straight lines called the polar plane. Wire. From a frame of images continuously shot by two image capture devices 30 in motion, it is assumed that the imaging points of the target point p in the two image capture devices 30 are respectively p1 and p2, given the coordinate value of p1, and using Based on the principle of motion recovery structure, that is, the farther the object is, the smaller the parallax is, and vice versa, the greater the parallax, we can measure the epipolar constraint parameter (Epipolar Constraint) in the polar geometry principle. The constraint parameter is the matching point p2 in 3D The direction of movement in space along the epipolar line.

Based on the above, the intelligent two-stage security control system for vehicles of the present invention controls the opening or closing of the car door through the cloud key exchange authentication and the two-stage authentication of the driver's face recognition by the convolutional neural network CN1, without the need for the driver to carry Metal keys or wafer keys.

The above descriptions are illustrative only, not restrictive. Any equivalent modification or change made without departing from the spirit and scope of the present invention shall be included in the scope of the appended patent application.

10: Electronic device 20: Servo processing calculator 21,311: processing units 22,312: Memory 23,313: transceiver unit 24:Warning unit 25: First database 26:Second database 27: Single multi-frame detection unit 30: Image capture device 31: Image receiving and transmitting processor 40: Cloud platform 41:Cloud database A: array AS: Authentication Procedure AG1: Object Recognition Algorithm AG2: Face Recognition Algorithm b: the distance between two image capture devices C: car CN1, CN2: Convolutional Neural Networks conv: convolutional layer D: driver DF: Driver Reference Feature EI: Environmental Imaging IS: video streaming f: focal length FI: partial image FL: fully connected layer MPL: Maximum Pooling Layer 𝑂1,𝑂2: Center points of two image capture devices P1: small image p1(x1,y1) ,p2(x2,y2): coordinates of a pair of image capture devices P(x,y,z): the coordinates of the truck RF: Reference facial features RO: Reference Object Characteristic R1: visual blind zone R2: Visual area SR: sensing area SS: similarity SCN1, SCN2: Preliminary Convolutional Neural Networks S11~S19: Process T: key TH1: Threshold value V: truck WLS: wireless signal

Fig. 1A is a block diagram of the vehicle intelligent two-stage safety control system of the present invention. Figure 1B is a schematic diagram of the use of the two-stage safety control system of the vehicle intelligence of the present invention. Fig. 2 is a flowchart of the cloud key exchange authentication of the present invention. Fig. 3 is a configuration diagram of the algorithm of the present invention. Fig. 4 is a flow chart of object recognition and face recognition of the present invention. Fig. 5 is a configuration diagram of the convolutional neural network of the present invention. Fig. 6 is a schematic diagram of the operation of the convolutional neural network of the present invention. Fig. 7 is a schematic diagram of the blind spot detection of the present invention. Fig. 8 is a schematic diagram of the measurement of the motion recovery structure of the present invention.

10: Electronic device

20: Servo processing calculator

21,311: processing units

22,312: Memory

23,313: transceiver unit

24:Warning unit

25: First database

26:Second database

27: Single multi-frame detection unit

30: Image capture device

31: Image receiving and transmitting processor

40: Cloud platform

41:Cloud database

AS: Authentication Procedure

AG1: Object Recognition Algorithm

AG2: Face Recognition Algorithm

CN1,CN2: Convolutional Neural Networks

EI: Environmental Imaging

IS: video streaming

SS: similarity

T: key

TH1: Threshold value

WLS: wireless signal

Claims (10)

A two-stage safety control system for vehicle intelligence, which includes: an electronic device; a pair of image capture devices, respectively installed in a rearview mirror of a vehicle; a pair of image receiving and transmitting processors, electrically connected to the pair Image capture device The pair of image receiving and transmitting processors includes an image receiving and transmitting processor processing unit, an image receiving and transmitting processor memory, and an image receiving and transmitting processor transceiver unit; a servo processing computing unit is embedded in the The vehicle is electrically connected to the pair of image receiving and transmitting processors. The servo processing unit includes a processing unit, a memory and a transceiver unit. The memory stores a convolutional neural network. When a driver carries the electronic When the device is close to a door of the car, the transceiver unit sends a wireless signal, the electronic device receives the wireless signal and completes a pairing operation with the servo processing unit, and the transceiver unit sends a key to the electronic device; and a The cloud platform connects the server processing computer and the electronic device through a network, the electronic device sends the key to a cloud platform for an authentication procedure, and the cloud platform sends the key to the server processing computer after passing the authentication procedure ; Wherein, when the server processing operator receives the key after passing the authentication procedure, the processing unit opens the rearview mirror and controls the pair of image capture devices and the pair of image receiving and transmitting processors to open, the pair of image The capture device shoots an environmental image around the driver, the pair of image receiving and transmitting processors converts the environmental image into an image stream and transmits it to the server processing processor through wireless transmission control protocol, the processing unit Using the convolutional neural network to compare the environment A driver's face in the image and a driver's reference image stored in the memory are used to calculate a similarity, and the processing unit executes a control action on the vehicle door according to the similarity. In the two-stage safety control system of vehicle intelligence described in claim 1, if the processing unit judges that the similarity is greater than a threshold value, the control action is to open the door. In the two-stage intelligent safety control system for vehicles as described in claim 1, if the processing unit judges that the similarity is less than a threshold value, the control action is to close the door. According to the two-stage security control system of vehicle intelligence described in claim 1, the servo processing operation unit further includes a warning unit, and when the servo processing operation unit receives the key that has not passed the authentication procedure, the processing unit controls The warning unit is activated. According to claim 1, when the driver carries the electronic device away from the car door, the transceiver unit does not receive the key, and the processing unit controls the car door to close. According to the two-stage safety control system of vehicle intelligence described in claim 1, the memory further stores an object recognition algorithm and a face recognition algorithm, and the processing unit uses the object recognition algorithm to identify a plurality of objects in the environment image object. As for the two-stage safety control system of vehicle intelligence described in claim 6, the servo processing unit further includes a first database and a second database, the cloud platform includes a cloud database, and the first data The database stores a plurality of driver reference features, the second database stores a plurality of reference object features, and the cloud The database stores a plurality of reference face features, the processing unit uses a preliminary convolutional neural network to initially identify the plurality of objects and the plurality of individuals in the environmental image, and the processing unit uses the object recognition algorithm to compare the The plurality of objects and the plurality of reference object features are used to identify the plurality of objects, and the processing unit uses the convolutional neural network and the face recognition algorithm to compare the plurality of individuals and the plurality of drivers in the environmental image respectively. The driver calculates a plurality of similarities with reference to features, and the processing unit selects one of the plurality of individuals as the face of the driver according to the plurality of similarities. In the two-stage safety control system for vehicle intelligence described in claim 7, when the processing unit recognizes the plurality of objects and the plurality of individuals, the processing unit calculates a first distance and a distance between the plurality of individuals and the vehicle. A second distance between the plurality of objects and the vehicle. According to the two-stage safety control system of vehicle intelligence described in claim 8, the processing unit judges that the plurality of objects and the plurality of individuals fall within a visible area or a distance according to the plurality of first distances and the plurality of second distances Visual blind spots. According to the two-stage safety control system of vehicle intelligence described in claim 6, the servo processing unit further includes a single multi-frame detection unit, and the single multi-frame detection unit utilizes a plurality of bounding boxes to the environment image Multiple individuals and multiple objects are framed separately.

Images