CN113524182B - Device and method for intelligently adjusting distance between person and screen - Google Patents
Device and method for intelligently adjusting distance between person and screen Download PDFInfo
- Publication number
- CN113524182B CN113524182B CN202110789453.7A CN202110789453A CN113524182B CN 113524182 B CN113524182 B CN 113524182B CN 202110789453 A CN202110789453 A CN 202110789453A CN 113524182 B CN113524182 B CN 113524182B
- Authority
- CN
- China
- Prior art keywords
- mechanical arm
- fatigue
- steering engine
- key points
- detection module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
- B25J13/087—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices for sensing other physical parameters, e.g. electrical or chemical properties
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J17/00—Joints
- B25J17/02—Wrist joints
- B25J17/0258—Two-dimensional joints
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J18/00—Arms
- B25J18/02—Arms extensible
- B25J18/025—Arms extensible telescopic
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
- B25J19/021—Optical sensing devices
- B25J19/023—Optical sensing devices including video camera means
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09F—DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS
- G09F9/00—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements
- G09F9/30—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements in which the desired character or characters are formed by combining individual elements
Abstract
The invention discloses a device and a method for intelligently adjusting the distance between a person and a screen. The device comprises a display screen, a camera sensor, a telescopic mechanical arm, an image recognition module, a fatigue detection module, a signal processing module and a driving motor controller which are connected; the telescopic mechanical arm is connected with the camera sensor, and the driving motor controller is used for controlling telescopic movement of the telescopic mechanical arm; the image recognition module is used for recognizing the existence state of a person in front of the screen and transmitting the recognized result into the driving motor controller; the fatigue detection module is used for identifying human body characteristics and behavior modes and judging whether a user is in a fatigue state or not; the driving motor controller receives signals transmitted back by the image recognition module and the fatigue detection module, and controls the action of the telescopic mechanical arm under the action of the built-in program.
Description
Technical Field
The invention relates to a display screen bracket, in particular to a display screen bracket capable of intelligently adjusting the distance between a person and a screen.
Background
Along with the continuous improvement of science and technology, electronic devices such as computers and televisions are commonly used in daily life of people and become a main medium for people to learn and understand external information, so that the installation and telescopic movement of the devices such as the computers and the televisions are a requirement of people. The screen adjusting devices in the market are all characterized in that the screen is arranged on a fixed mounting frame, the position of the screen can be adjusted only by manual operation, and no mechanical device can automatically stretch and retract to adjust the front and back displacement of the screen. Therefore, there is an urgent need to develop a device and a method for intelligently adjusting the distance between a person and a screen, which can automatically adjust the front-back displacement of the screen in the horizontal direction conveniently, and detect the existence state and the gesture characteristics of the person in real time so as to realize intelligent movement.
Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a device and a method for intelligently adjusting the distance between a person and a screen, which can effectively solve the problem that all supports on the market are manually adjusted, effectively liberate hands of the person, and further improve the working efficiency of the person.
The technical scheme of the invention is as follows: the device for intelligently adjusting the distance between the person and the screen comprises a display screen, and a camera sensor, a telescopic mechanical arm, an image recognition module, a fatigue detection module, a signal processing module and a driving motor controller which are connected. The telescopic mechanical arm is connected with the camera sensor, and the driving motor controller is used for controlling telescopic movement of the telescopic mechanical arm; the image recognition module is used for recognizing the existence state of a person in front of the screen and transmitting the recognized result into the driving motor controller; the fatigue detection module is used for identifying human body characteristics and behavior modes and judging whether a user is in a fatigue state or not; the driving motor controller receives signals transmitted back by the image recognition module and the fatigue detection module, and controls the action of the telescopic mechanical arm under the action of the built-in program and the signal processing module.
The driving motor controller comprises a digital steering engine 11, a first serial port steering engine 12 and a second serial port steering engine 13; the digital steering engine 11 adopts an MG995 type digital steering engine, and the first serial port steering engine 12 and the second serial port steering engine 13 both adopt MG996R type serial port motors; the central processor of the driving motor controller controls the movement of the steering engine according to the length of the clock pulse signal; the digital steering engine 11 simultaneously controls a first serial port steering engine 12 and a second serial port steering engine 13; the two serial port steering engines are used for controlling the action of the telescopic mechanical arm, and the digital steering engine 11 is used for realizing the direction adjustment of the telescopic mechanical arm.
The telescopic mechanical arm comprises a base 1, a positioning seat 2, a rotary table 3, a first mechanical arm 4 of a main mechanical arm, a second mechanical arm 5 of the main mechanical arm, a triangular rotary plate 6, a secondary mechanical arm 7, a first connecting rod 8, a crank 9, a rotary gear 14, a second connecting rod 15 and a third connecting rod 16.
The base 1 is embedded with the positioning seat 2, the digital steering engine 11 is positioned in the base 1, the digital steering engine comprises a programmable logic chip, and PWN control signals are sent to the first serial port steering engine 12 and the second serial port steering engine 13 according to a set program.
The rotary table 3 is fastened with the first mechanical arm 4 of the main mechanical arm, a first serial port steering engine 12 and a second serial port steering engine 13 are respectively arranged on the left side and the right side of the rotary table 3, and the two serial port steering engines are connected with the digital steering engine 11 through wires; the first serial steering engine 12 is used for controlling the motion and the angle of the first mechanical arm 4 of the main mechanical arm, and simultaneously provides a feedback signal for the digital steering engine through current and voltage signals.
The first mechanical arm 4 of the main mechanical arm is connected with a triangular rotating plate 6 at one end far away from the rotating table; the second serial port steering engine 13 is used for controlling the movement and the angle of the second mechanical arm 5 of the main mechanical arm; the second mechanical arm 5 of the main mechanical arm is connected with a crank 9; the upper end of the crank 9 is riveted with the first connecting rod 8 through a bearing, and the crank 9 drives the first connecting rod 8 to move.
The triangular rotating plate 6 is provided with two ends which are respectively hinged with the auxiliary mechanical arm 7 and the third connecting rod 16, and the triangular rotating plate 6 is used for adjusting the angle so as to ensure that the auxiliary mechanical arm 7 and the third connecting rod 16 stably stretch and retract the display 10 in the moving process; one end of the auxiliary mechanical arm 7 and the second connecting rod 15, which is far away from the triangular turning piece 6, is buckled with the display 10 to adjust the display direction.
The image recognition module works according to the following modes under the action of a built-in program:
judging whether people or other objects appear in front of the camera by adopting a pretrained VGG16 convolutional neural network, once the people appear in front of the camera is determined, processing an image acquired by the camera in real time into 128 pixels in a background
128 pixel images, 16384 pixels total, if identified, of the user's faceThe occupied size is more than 30 pixels +.>
The face of a user in front of the camera is monitored by 30 pixels or more than 900 pixels, and a face fatigue detection part in the fatigue detection module is activated; if the size of the face of the identified user is less than 30 pixels +.>
The whole body/half body monitoring is carried out on users in front of the camera by 30 pixels or less than 900 pixels, and a whole body/half body fatigue detection part in the fatigue detection module is activated; after the corresponding fatigue detection module is determined to be activated, judging the brightness of the environment according to the Laplace edge detection theory, namely 128 pixels in size are added in the background>
The 128-pixel image is converted into a gray-scale image for which 3 +.>
3, convolving the Laplacian operator to obtain an edge detection diagram and calculating variance of the edge detection diagram, wherein the environment is defined to be in a bright state when the variance of the edge detection diagram is larger than 600, and the environment is defined to be in a dark state when the variance of the edge detection diagram is smaller than 600; and constructing the information of the activation module and the ambient light and shade information into a matrix, and then transmitting the matrix into the fatigue detection module.
The fatigue detection module consists of an LSTM long and short memory neural network, a GAN countermeasure neural network and an OpenCV and OpenPose, wherein the LSTM long and short memory neural network, the GAN countermeasure neural network and the OpenCV and the OpenPose are the face fatigue detection part in the fatigue detection module, and the OpenCV and the OpenPose are the whole body/half body fatigue detection part in the fatigue detection module; the fatigue detection module operates in the following mode under the action of a built-in program.
When the fatigue detection module receives the face detection activation information and the environment brightness information matrix transmitted by the image recognition module, only activating an LSTM long and short memory neural network to observe 68 key points of the face of a user, firstly counting blinks of eye key points, establishing a fatigue judgment formula by comparing the relationship between blink frequency and image frame number, and taking the eye key points as first fatigue judgment, secondly positioning the mouth key points, judging whether the user is yawing and records yawing time according to the opening and closing degree of the mouth, taking the yawing time as second fatigue judgment, finally positioning the head key points, taking the key point of the chin as a normal point, establishing a perpendicular bisector, and calculating the included angle between the face and the perpendicular bisector as third fatigue judgment; when the fatigue detection module receives the face detection activation information and the environment darkness information matrix transmitted by the image recognition module, activating a GAN (gas-water-gas) anti-neural network at the moment, recovering the details of the five sense organs based on a pre-trained generation model, restoring the low-quality face data obtained by the camera into a high-quality image, and then carrying out the fatigue judgment on 68 key points of the face of a user by utilizing the LSTM long and short memory neural network; when the fatigue detection module receives whole body/half body detection activation information transmitted back by the image recognition module, activating an OpenCV+OpenPose part, extracting 1 head key points S1,2 shoulder key points S2 and S3,2 hand ankle key points S4 and S5,2 crotch key points S6 and S7,2 knee key points S8 and S9 and 2 ankle key points S10 and S11 from an input real-time image through a convolutional neural network, taking Euclidean distances of the head key points S1 and the shoulder key points S2 and S3 as first fatigue judgment, taking Euclidean distances of the head key points S1 and the hand ankle key points S4 and S5 as second fatigue judgment, and taking Manhattan distances of the 2 knee key points S8 and S9 as third fatigue judgment; and finally, classifying the states of whether fatigue exists into two types, assigning the fatigue value to be 1 and the fatigue value not to be 0, and transmitting the fatigue information to the signal processing module in a signal mode of 1 or 0.
The signal processing module is used for receiving the 1 or 0 signal transmitted back by the image recognition module and the fatigue detection module and converting the signal into a driving electric signal readable by the driving motor controller, and commanding the driving motor controller to enter a working state.
The invention has the following beneficial effects: when a user does not work or watch the screen, the mechanical device for intelligently adjusting the screen is not in a working state, and the telescopic mechanical arm is in a dormant state; when the image recognition module and the fatigue detection module monitor that a user works or watches before the screen, the mechanical device for intelligently adjusting the screen enters a working state, image data is immediately transmitted into the signal processing module to be extracted and converted into an effective readable format to be transmitted into the driving motor controller, an electric signal command is sent out by the driving motor controller to be transmitted to the telescopic mechanical arm to enable the telescopic mechanical arm to conduct horizontal front-back adjustment on the screen, when the mechanical device for intelligently adjusting the screen is in the working state, the mechanical device for intelligently adjusting the screen observes that the user continuously exists before the display screen for 30 minutes, the driving motor controller and the telescopic mechanical arm start to automatically push and pull the screen to conduct horizontal front-back movement, and the horizontal front-back movement is conducted once in a range of 5-15 minutes after starting work according to the fatigue state of the user.
Description of the drawings:
FIG. 1 is a flowchart of an image recognition module according to an embodiment of the present invention;
FIG. 2 is a flowchart of a fatigue detection module according to an embodiment of the present invention;
FIG. 3 is a schematic view of an appearance structure according to an embodiment of the present invention;
FIG. 4 is an exploded view of an exterior structure according to an embodiment of the present invention;
FIG. 5 is a graph of 68 fatigue detection keypoints locations according to an embodiment of the invention;
fig. 6 is a diagram of 11 identified node locations in accordance with an embodiment of the present invention.
The specific embodiment is as follows:
the invention is further described below with reference to the accompanying drawings:
FIG. 1 is a flowchart of an image recognition module of an intelligent person-to-screen distance adjusting device, which is implemented according to the flowchart of the method, namely, a pretrained VGG16 convolutional neural network is adopted after a camera sensor receives an image, whether a person or other objects appear in front of a camera is judged, and once the person appearing in front of the camera is determined, the image recognition module is started, and the image recognition module is startedAfter program initialization, images acquired by the receiving cameras in real time are processed into 128-pixel images in the background
128-pixel image, 16384 pixels in total, if the size of the face of the identified user is greater than 30 pixels +.>
30 pixels or more than 900 pixels, the face fatigue detection part in the fatigue detection module is activated; if the size of the face of the identified user is less than 30 pixels +.>
The method comprises the steps that 30 pixels or less than 900 pixels are used for activating a whole body/half body fatigue detection part in a fatigue detection module, and activating information is assigned to be 1, and non-activating information is assigned to be 0; the Laplace edge detection is carried out while the face pixel points are counted, and the brightness of the environment is judged, namely 128 pixels in size are judged in the background>
The 128-pixel image is converted into a gray-scale image for which 3 +.>
3 convolving the Laplacian operator to obtain an edge detection diagram and calculating variance on the edge detection diagram, defining the environment as a bright state when the variance of the edge detection diagram is larger than 600, defining the environment as a dark state when the variance of the edge detection diagram is smaller than 600, assigning the environment bright information as 1 and the environment dark information as 0, constructing a matrix, and transmitting the matrix into a fatigue detection module, wherein the face detection matrix when the environment is bright is [0,1, 0]]The method comprises the steps of carrying out a first treatment on the surface of the The whole body/half body detection matrix in dark environment is [1,0,1]。
FIG. 2 is a flowchart of a fatigue detection module, and the device is implemented according to the flowchart of the method, namely when the fatigue detection module receives the information matrix [0,1, 0] transmitted by the image recognition module, only an LSTM long and short memory neural network is activated to observe 68 key points of the face of a user, the 68 key points are shown in FIG. 5, firstly, eye key points are subjected to blink counting, a fatigue judgment formula is established by comparing the relationship between blink frequency and image frame number to be used as a first fatigue judgment, secondly, a mouth key point is positioned, whether the user is yawed and yawed time is recorded according to the opening and closing degree of the mouth to be used as a second fatigue judgment, finally, the head key point is positioned, a perpendicular bisector is established by taking the key point of the chin part as a normal point, and the included angle between the face and the perpendicular bisector is calculated to be used as a third fatigue judgment; when the fatigue detection module receives the information matrix [1,0,1,0] transmitted by the image recognition module, activating a GAN (gas barrier network), recovering the five sense organ details based on a pre-trained generation model, recovering the low-quality face data obtained by the camera into a high-quality image, and then carrying out the fatigue judgment on 68 key points of the face of a user by utilizing the LSTM long and short memory neural network;
when the fatigue detection module receives the information matrix transmitted by the image recognition module as [0,1,0,1] or [1,0, 1], an opencv+openPose part is activated, an input real-time image is subjected to convolutional neural network to extract a human skeleton frame, as shown in fig. 6, 1 head key point S1,2 shoulder key points S2 and S3,2 hand ankle key points S4 and S5,2 crotch key points S6 and S7,2 knee key points S8 and S9 and 2 ankle key points S10 and S11, euclidean distance between the head key point S1 and the shoulder key points S2 and S3 is used as a first fatigue judgment, euclidean distance between the head key point S1 and the hand ankle key points S4 and S5 is used as a second fatigue judgment, manhattan distance between the 2 knee key points S8 and S9 is used as a third fatigue judgment, and finally whether the telescopic mechanical arm is started or not is judged according to whether the fatigue judgment conditions are met.
Fig. 3 and 4 are an appearance diagram and an appearance explosion diagram of a device for intelligently adjusting the distance between a person and a screen, which comprise a camera sensor, a telescopic mechanical arm, an image recognition module, a fatigue detection module, a signal processing module and a driving motor controller which are connected; the driving motor controller is respectively associated with the camera sensor, the telescopic mechanical arm, the image recognition module, the fatigue detection module and the signal processing module, when a user appears in front of the camera sensor 17, the image recognition module and the fatigue detection module monitor that the user, the mechanical device of the intelligent adjusting screen enters a working state, immediately transmits image data into the signal processing module to extract and convert the image data into an effective readable format, transmits the effective readable format into the driving motor controller, and then transmits an electric signal command to the telescopic mechanical arm by the driving motor controller, and simultaneously adjusts the horizontal distance and the angle of the screen; when the bracket extends forwards, the serial port steering engine 12 performs lower swinging motion together with the first mechanical arm 4 of the main mechanical arm and the second mechanical arm 3 of the main mechanical arm through the rotary gear, and the crank 9 drives the first connecting rod 8 to press the second connecting rod 15 to swing upwards so as to push the screen to move forwards. The triangular rotating piece connected with the No. 2 connecting rod further rotates towards the front direction of the screen to drive the triangular rotating piece 6 and the third connecting rod 16 connected with the other end to do a lower swinging motion so that the mechanical arm reaches a stretching state, the connecting rod 16 regulates the movement of the triangular rotating piece and the third connecting rod 16, when the main mechanical arm further swings downwards, the triangular rotating piece connected with the triangular rotating piece rotates clockwise to support the auxiliary mechanical arm to do an upper swinging motion through a bearing, meanwhile, the second connecting rod 15 also does the same motion so that the main mechanical arm and the auxiliary mechanical arm gradually approach to a straightening state, the purpose of reducing the distance between the screen and a person is achieved, the distance regulation is achieved, the triangular rotating piece 6 in an initial state and the third connecting rod 16 of the mechanical arm are set with a certain angle, the height of the screen is not changed in the distance regulation process, and the reverse direction change of the movement state is shown when the support is compressed backwards; the angle adjustment, the rotating table 3 of the mechanical arm base part can meet the function of the mechanical arm for freely rotating and adjusting the angle. The digital steering engine 11 connected to the bottom of the mechanical arm is used as two main driving forces with a worm in the rotating base, and when a signal instruction is received, the digital steering engine 11 connected to the bottom of the mechanical arm drives the rotating shaft to control the mechanical arm to rotate in a worm transmission mode, so that the aim of adjusting the angle between the screen and a person is fulfilled.
The above drawings are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In addition, the shapes, proportional sizes, and the like of the respective components in the drawings are merely illustrative for aiding the understanding of the present application, and are not particularly limited. Those skilled in the art who have the benefit of the teachings of this application may select various possible shapes and scale dimensions to practice this application as the case may be.
In summary, the device and the method for intelligently adjusting the distance between a person and a screen comprise a camera sensor, a telescopic mechanical arm, an image recognition module, a fatigue detection module, a signal processing module and a driving motor controller which are connected; the driving motor controller is respectively associated with the camera sensor, the telescopic mechanical arm, the image recognition module, the fatigue detection module and the signal processing module.
The camera sensor is mainly a camera or an independent camera of a display screen; the telescopic mechanical arm is connected with the camera sensor, and the telescopic movement of the module is controlled by the driving motor controller.
The telescopic mechanical arm structurally comprises a base 1, a positioning seat 2, a rotary table 3, a first mechanical arm 4 of a main mechanical arm, a second mechanical arm 5 of the main mechanical arm, a triangular rotary plate 6, an auxiliary mechanical arm 7, a first connecting rod 8, a crank 9, a display screen 10, a digital steering engine 11, two serial port steering engines 12 and 13, a rotary gear 14, a second connecting rod 15 and a third connecting rod 16; the base 1 is embedded with the positioning seat 2, a digital steering engine 11 is arranged in the base 1, a programmable logic chip is arranged in the digital steering engine, and a PWN control signal is sent to the serial port steering engine according to a set program; the positioning seat 2 is connected with the base 1 and the rotary table 3, the rotary table 3 is buckled with the main mechanical arm large arm 4, serial port steering gears 12 and 13 are arranged on the left side and the right side of the rotary table 3, the two serial port steering gears are connected with the digital steering gear 11 in the base through wires, the serial port steering gear 12 is used for controlling the movement and the angle of the main mechanical arm first mechanical arm 4, and simultaneously, feedback is provided for the digital steering gear in a current and voltage mode; the first mechanical arm 4 of the main mechanical arm is connected with the triangular rotating plate 6 at one end far away from the rotating table, the serial steering engine 13 controls the movement and the angle of the second mechanical arm 5 of the main mechanical arm, the second mechanical arm 5 of the main mechanical arm is connected with the crank 9 at one end close to the serial steering engine 5, the upper end of the crank 9 is riveted with the first connecting rod 8 through a bearing, and the crank 9 drives the first connecting rod 8 to move; the other two ends of the triangular rotating piece 6 are respectively hinged with the auxiliary mechanical arm 7 and the third connecting rod 16, and the angle of the triangular rotating piece 6 is adjusted to ensure that the auxiliary mechanical arm 7 and the third connecting rod 16 stably stretch and retract the display in the moving process; one end of the auxiliary mechanical arm 7 and the second connecting rod 15, which is far away from the triangular turning piece 6, is connected with the movable transposition; the movable device is buckled with the display 10, and a user can mobilize the movable device according to the requirement to change the direction of the display.
The image recognition module adopts a pretrained VGG16 convolutional neural network to judge whether people or other objects appear in front of the camera, and once the people appear in front of the camera is determined, the image acquired by the camera in real time is processed into 128-pixel images in the background
128-pixel image, 16384 pixels in total, if the size of the face of the identified user is greater than 30 pixels +.>
The face of a user in front of the camera is monitored by 30 pixels or more than 900 pixels, and a face fatigue detection part in the fatigue detection module is activated; if the size of the face of the identified user is less than 30 pixels +.>
The whole body/half body monitoring is carried out on users in front of the camera by 30 pixels or less than 900 pixels, and a whole body/half body fatigue detection part in the fatigue detection module is activated; after the corresponding fatigue detection module is determined to be activated, judging the brightness of the environment, wherein the main basis of the judgment is Laplacian edge detection theory, and 128 pixels are used as the size in the background>
The 128-pixel image is converted into a gray-scale image for which 3 +.>
3, convolving the Laplacian operator to obtain an edge detection diagram and calculating variance on the edge detection diagram, defining the environment to be in a bright state when the variance of the edge detection diagram is larger than 600, defining the environment to be in a dark state when the variance of the edge detection diagram is smaller than 600, constructing the activation module information and the environment brightness information into a matrix, and then transmitting the matrix into the fatigue detection module.
The fatigue detection module consists of an LSTM long and short memory neural network, a GAN countermeasure neural network and an OpenCV and OpenPose, wherein the LSTM long and short memory neural network, the GAN countermeasure neural network and the OpenCV and the OpenPose are the facial fatigue detection part in the fatigue detection module, and the OpenCV and the OpenPose are the whole body/half body fatigue detection part in the fatigue detection module; when the fatigue detection module receives the face detection activation information and the environment brightness information matrix transmitted by the image recognition module, only activating an LSTM long and short memory neural network to observe 68 key points of the face of a user, firstly counting blinks of eye key points, establishing a fatigue judgment formula by comparing the relationship between blink frequency and image frame number, and taking the eye key points as first fatigue judgment, secondly positioning the mouth key points, judging whether the user is yawing and records yawing time according to the opening and closing degree of the mouth, taking the yawing time as second fatigue judgment, finally positioning the head key points, taking the key point of the chin as a normal point, establishing a perpendicular bisector, and calculating the included angle between the face and the perpendicular bisector as third fatigue judgment; when the fatigue detection module receives the face detection activation information and the environment darkness information matrix transmitted by the image recognition module, activating a GAN (gas-water-gas) anti-neural network at the moment, recovering the details of the five sense organs based on a pre-trained generation model, restoring the low-quality face data obtained by the camera into a high-quality image, and then carrying out the fatigue judgment on 68 key points of the face of a user by utilizing the LSTM long and short memory neural network; when the fatigue detection module receives whole body/half body detection activation information transmitted back by the image recognition module, activating an OpenCV+OpenPose part, extracting 1 head key points S1,2 shoulder key points S2 and S3,2 hand ankle key points S4 and S5,2 crotch key points S6 and S7,2 knee key points S8 and S9 and 2 ankle key points S10 and S11 from an input real-time image through a convolutional neural network, taking Euclidean distances of the head key points S1 and the shoulder key points S2 and S3 as first fatigue judgment, taking Euclidean distances of the head key points S1 and the hand ankle key points S4 and S5 as second fatigue judgment, and taking Manhattan distances of the 2 knee key points S8 and S9 as third fatigue judgment; finally, whether the fatigue state is classified into two types, the fatigue is assigned to be 1, the fatigue is assigned to be 0, and then the fatigue information is transmitted to the signal processing module in a signal mode of 1 or 0.
The signal processing module and the driving motor controller are used for receiving the 1 or 0 signal transmitted back by the image recognition module and the fatigue detection module and converting the signal into a driving electric signal readable by the driving motor controller, and commanding the driving motor controller to enter a working state; the driving motor controller consists of a digital steering engine 11 of an MG995 model and serial motors 12 and 13 of two MG996R, a central processor of the driving motor controller is positioned in the digital steering engine in the base 1, the central processor controls the movement of the steering engine according to the length of clock pulse signals, the central processor further sends out signals to the steering engine through judgment of the fatigue detection module, and a driving circuit in the steering engine drives the motor to drive the mechanical arm to move after receiving the signals. The digital steering engine 11 simultaneously controls two serial port steering engines 12 and 13, the extension and retraction of the mechanical arm are mainly completed by the two serial port steering engines, and the direction adjustment of the mechanical arm is completed by the digital steering engine 11.
As a further improvement of the invention, when the device for intelligently adjusting the distance between a person and the screen is in a working state, and the situation that the user continuously exists in front of the display screen for 30 minutes is observed, the driving motor controller and the telescopic mechanical arm start to automatically push and pull the screen to move back and forth in the horizontal direction, and after the starting work, the device moves back and forth in the horizontal direction once in the interval of 5-15 minutes according to the fatigue state of the user.
As a further improvement of the invention, the device for intelligently adjusting the distance between the person and the screen monitors that the person is in a working state immediately after 30 minutes before the person appears on the display screen; the mechanical device of the intelligent regulation screen immediately stops working state after monitoring that a person leaves the display screen for 5 minutes, so that the working efficiency of the mechanical device of the intelligent regulation screen is ensured.
As a further improvement of the invention, the device for intelligently adjusting the distance between the person and the screen realizes the switching of the intelligent adjusting telescopic mechanical arm and the manual adjusting telescopic mechanical arm on a software level.
As a further improvement of the invention, the device for intelligently adjusting the distance between the person and the screen adopts a Bluetooth/WIFI connection mode, and the automatic and manual gear of the telescopic mechanical arm is controlled through a mobile phone App.
Claims (2)
1. The utility model provides a device of intelligent regulation people and screen distance, includes display screen and the camera sensor of connection, its characterized in that:
the device also comprises a telescopic mechanical arm, an image recognition module, a fatigue detection module, a signal processing module and a driving motor controller; the telescopic mechanical arm is connected with the camera sensor, and the driving motor controller is used for controlling telescopic movement of the telescopic mechanical arm; the image recognition module is used for recognizing the existence state of a person in front of the screen and transmitting the recognized result into the driving motor controller; the fatigue detection module is used for identifying human body characteristics and behavior modes and judging whether a user is in a fatigue state or not; the driving motor controller receives signals transmitted back by the image recognition module and the fatigue detection module, and controls the action of the telescopic mechanical arm under the action of the built-in program and the signal processing module;
the driving motor controller comprises a digital steering engine (11), a first serial port steering engine (12) and a second serial port steering engine (13); the digital steering engine (11) adopts an MG995 type digital steering engine, and the first serial port steering engine (12) and the second serial port steering engine (13) both adopt MG996R type serial port motors; the central processor of the driving motor controller controls the movement of the steering engine according to the length of the clock pulse signal; the digital steering engine (11) simultaneously controls the first serial port steering engine (12) and the second serial port steering engine (13); the two serial port steering engines are used for controlling the action of the telescopic mechanical arm, and the digital steering engine (11) is used for realizing the direction adjustment of the telescopic mechanical arm;
the telescopic mechanical arm comprises a base (1), a positioning seat (2), a rotary table (3), a first mechanical arm (4) of a main mechanical arm, a second mechanical arm (5) of the main mechanical arm, a triangular rotary piece (6), a secondary mechanical arm (7), a first connecting rod (8), a crank (9), a rotary gear (14), a second connecting rod (15) and a third connecting rod (16);
the base (1) is embedded with the positioning seat (2), the digital steering engine (11) is positioned in the base (1), a programmable logic chip is contained in the digital steering engine, and PWN control signals are sent to the first serial port steering engine (12) and the second serial port steering engine (13) according to a set program;
the rotary table (3) is buckled with the first mechanical arm (4) of the main mechanical arm, a first serial port steering engine (12) and a second serial port steering engine (13) are respectively arranged at the left side and the right side of the rotary table (3), and the two serial port steering engines are connected with the digital steering engine (11) through electric wires; the first serial port steering engine (12) is used for controlling the movement and the angle of the first mechanical arm (4) of the main mechanical arm and simultaneously providing a feedback signal for the digital steering engine through current and voltage signals;
the first mechanical arm (4) of the main mechanical arm is connected with a triangular rotating plate (6) at one end far away from the rotating table; the second serial port steering engine (13) is used for controlling the movement and the angle of a second mechanical arm (5) of the main mechanical arm; the second mechanical arm (5) of the main mechanical arm is connected with a crank (9); the upper end of the crank (9) is riveted with the first connecting rod (8) through a bearing, and the first connecting rod (8) is driven to move through the crank (9);
the triangular rotating piece (6) is provided with two ends which are respectively hinged with the auxiliary mechanical arm (7) and the third connecting rod (16), and the triangular rotating piece (6) is used for adjusting the angle so as to ensure that the auxiliary mechanical arm (7) and the third connecting rod (16) stably stretch and retract the display (10) in the moving process; one end, far away from the triangular rotating piece (6), of the auxiliary mechanical arm (7) and the second connecting rod (15) is buckled with the display (10) to adjust the direction of the display;
the image recognition module works according to the following modes under the action of a built-in program:
VGG16 roll with pretrainedThe neural network is used for judging whether people or other objects appear in front of the camera, and once the people appear in front of the camera is determined, the image acquired by the camera in real time is processed into 128-pixel images in the background
128-pixel image, 16384 pixels in total, if the size of the face of the identified user is greater than 30 pixels +.>
The face of a user in front of the camera is monitored by 30 pixels or more than 900 pixels, and a face fatigue detection part in the fatigue detection module is activated; if the size of the face of the identified user is less than 30 pixels +.>
The whole body/half body monitoring is carried out on users in front of the camera by 30 pixels or less than 900 pixels, and a whole body/half body fatigue detection part in the fatigue detection module is activated; after the corresponding fatigue detection module is determined to be activated, judging the brightness of the environment according to the Laplace edge detection theory, namely 128 pixels in size are added in the background>
The 128-pixel image is converted into a gray-scale image for which 3 +.>
3, convolving the Laplacian operator to obtain an edge detection diagram and calculating variance of the edge detection diagram, wherein the environment is defined to be in a bright state when the variance of the edge detection diagram is larger than 600, and the environment is defined to be in a dark state when the variance of the edge detection diagram is smaller than 600; the information of the activation module and the ambient light and shade information are constructed into a matrix and then are transmitted into a fatigue detection module;
the fatigue detection module consists of an LSTM long and short memory neural network, a GAN countermeasure neural network and an OpenCV and OpenPose, wherein the LSTM long and short memory neural network, the GAN countermeasure neural network and the OpenCV and the OpenPose are the face fatigue detection part in the fatigue detection module, and the OpenCV and the OpenPose are the whole body/half body fatigue detection part in the fatigue detection module; the fatigue detection module works under the action of a built-in program according to the following modes:
when the fatigue detection module receives the face detection activation information and the environment brightness information matrix transmitted by the image recognition module, only activating an LSTM long and short memory neural network to observe 68 key points of the face of a user, firstly counting blinks of eye key points, establishing a fatigue judgment formula by comparing the relationship between blink frequency and image frame number, and taking the eye key points as first fatigue judgment, secondly positioning the mouth key points, judging whether the user is yawing and records yawing time according to the opening and closing degree of the mouth, taking the yawing time as second fatigue judgment, finally positioning the head key points, taking the key point of the chin as a normal point, establishing a perpendicular bisector, and calculating the included angle between the face and the perpendicular bisector as third fatigue judgment; when the fatigue detection module receives the face detection activation information and the environment darkness information matrix transmitted by the image recognition module, activating a GAN (gas-water-gas) anti-neural network at the moment, recovering the details of the five sense organs based on a pre-trained generation model, restoring the low-quality face data obtained by the camera into a high-quality image, and then carrying out the fatigue judgment on 68 key points of the face of a user by utilizing the LSTM long and short memory neural network; when the fatigue detection module receives whole body/half body detection activation information transmitted back by the image recognition module, activating an OpenCV+OpenPose part, extracting 1 head key points S1,2 shoulder key points S2 and S3,2 hand ankle key points S4 and S5,2 crotch key points S6 and S7,2 knee key points S8 and S9 and 2 ankle key points S10 and S11 from an input real-time image through a convolutional neural network, taking Euclidean distances of the head key points S1 and the shoulder key points S2 and S3 as first fatigue judgment, taking Euclidean distances of the head key points S1 and the hand ankle key points S4 and S5 as second fatigue judgment, and taking Manhattan distances of the 2 knee key points S8 and S9 as third fatigue judgment; finally, classifying the fatigue state into two types, assigning a fatigue value of 1 and a non-fatigue value of 0, and transmitting the fatigue information to the signal processing module in a signal mode of 1 or 0;
the signal processing module is used for receiving the 1 or 0 signal transmitted back by the image recognition module and the fatigue detection module and converting the signal into a driving electric signal readable by the driving motor controller, and commanding the driving motor controller to enter a working state.
2. A method for intelligently adjusting the distance between a person and a screen, using the device of claim 1, characterized in that: when the user is observed to continuously exist in front of the display screen for 30 minutes, the motor controller and the telescopic mechanical arm are driven to automatically push and pull to display under the control of the built-in program to move back and forth in the horizontal direction, and after the work is started, the user moves back and forth in the horizontal direction once in the interval of 5-15 minutes according to the fatigue state of the user; and stopping working state immediately after the person is monitored to leave the display screen for 5 minutes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110789453.7A CN113524182B (en) | 2021-07-13 | 2021-07-13 | Device and method for intelligently adjusting distance between person and screen |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110789453.7A CN113524182B (en) | 2021-07-13 | 2021-07-13 | Device and method for intelligently adjusting distance between person and screen |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113524182A CN113524182A (en) | 2021-10-22 |
| CN113524182B true CN113524182B (en) | 2023-05-16 |
Family
ID=78127667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110789453.7A Active CN113524182B (en) | 2021-07-13 | 2021-07-13 | Device and method for intelligently adjusting distance between person and screen |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113524182B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014158055A1 (en) * | 2013-03-25 | 2014-10-02 | Rytik Andrei Petrovich | Method for examining and assessing the eye fatigue of a personal computer user |
| KR20150139745A (en) * | 2014-06-04 | 2015-12-14 | 윤재웅 | Processing method for using smart phone closely to screen |
| CN107775658A (en) * | 2016-08-24 | 2018-03-09 | 南京乐朋电子科技有限公司 | Adapt to human comfort automatically adjust screen and face apart from robot |
| JP2018128640A (en) * | 2017-02-10 | 2018-08-16 | 富士ゼロックス株式会社 | Information processing apparatus, information processing system, and program |
| CN110096957A (en) * | 2019-03-27 | 2019-08-06 | 苏州清研微视电子科技有限公司 | The fatigue driving monitoring method and system merged based on face recognition and Activity recognition |
| CN110119672A (en) * | 2019-03-26 | 2019-08-13 | 湖北大学 | A kind of embedded fatigue state detection system and method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3705018A4 (en) * | 2017-11-01 | 2020-10-14 | Sony Corporation | Surgical arm system and surgical arm control system |
-
2021
- 2021-07-13 CN CN202110789453.7A patent/CN113524182B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014158055A1 (en) * | 2013-03-25 | 2014-10-02 | Rytik Andrei Petrovich | Method for examining and assessing the eye fatigue of a personal computer user |
| KR20150139745A (en) * | 2014-06-04 | 2015-12-14 | 윤재웅 | Processing method for using smart phone closely to screen |
| CN107775658A (en) * | 2016-08-24 | 2018-03-09 | 南京乐朋电子科技有限公司 | Adapt to human comfort automatically adjust screen and face apart from robot |
| JP2018128640A (en) * | 2017-02-10 | 2018-08-16 | 富士ゼロックス株式会社 | Information processing apparatus, information processing system, and program |
| CN110119672A (en) * | 2019-03-26 | 2019-08-13 | 湖北大学 | A kind of embedded fatigue state detection system and method |
| CN110096957A (en) * | 2019-03-27 | 2019-08-06 | 苏州清研微视电子科技有限公司 | The fatigue driving monitoring method and system merged based on face recognition and Activity recognition |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113524182A (en) | 2021-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104065862B (en) | A kind of information processing method and electronic equipment | |
| CN104331149B (en) | A kind of control method, device and electronic equipment | |
| CN109571513B (en) | Immersive mobile grabbing service robot system | |
| EP2877909A1 (en) | Multimodal interaction with near-to-eye display | |
| CN101989126A (en) | Handheld electronic device and automatic screen picture rotating method thereof | |
| CN102323829A (en) | Display screen visual angle regulating method and display device | |
| CN207586945U (en) | Face recognition device with light adjusting and wiping function | |
| CN108298084A (en) | A kind of armed unmanned plane of autonomous crawl object | |
| CN108989653A (en) | The fatigue driving early-warning device of vehicular adaptive environment light and head pose | |
| CN113524182B (en) | Device and method for intelligently adjusting distance between person and screen | |
| CN101615033A (en) | The angular adjustment apparatus of display module and method | |
| CN106214163B (en) | Recovered artifical psychological counseling device of low limbs deformity correction postoperative | |
| CN207382412U (en) | A kind of Booknote rotatable camera | |
| CN111726921A (en) | Somatosensory interactive light control system | |
| CN102316260A (en) | Control system and method of PTZ (pan/tilt/zoom) camera and adjusting device of PTZ camera | |
| CN206388249U (en) | A kind of intelligent rod-pulling type wireless video inspection device | |
| CN103124336A (en) | Television with angle adjustable built-in camera | |
| CN112466017A (en) | Intelligent conference security check equipment based on face recognition | |
| CN111336644A (en) | Air conditioner adjusting system based on eyeball drive control | |
| CN211698883U (en) | Gesture recognition control switch device based on FPGA | |
| CN216619205U (en) | Face recognition device capable of adjusting angle of face recognition instrument | |
| CN109348091A (en) | A kind of mobile phone camera | |
| CN112097055A (en) | Display that intelligence goes up and down and rotate and control system thereof | |
| CN204215113U (en) | Eye dynamic auxiliary device, eye move tracker and system controller | |
| CN108594861B (en) | Camera placing and moving device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |