KR20210115842A - Moving object and operation method with artificial intelligence vision processing and user tracking - Google Patents

Abstract

The present invention relates to a method for operating a moving object that follows a person using an artificial intelligence image processing. Described is an operating method of a camera and a moving object that recognizes a user from an optical camera image frame, and follows the user after extracting the data. Specifically, the present invention relates to a technology that allows a mobile device equipped with an artificial intelligence algorithm to continuously monitor the user, and allows the user to continuously exchange without missing out by operating in real-time in response to a movement of the user.

Description

{Moving object and operation method with artificial intelligence vision processing and user tracking}

The present invention relates to a method of controlling the movement of a camera and a moving object that follows a user in an optical camera image using artificial intelligence image processing. In more detail, it relates to a technology in which a mobile device equipped with an artificial intelligence algorithm maintains the user while not missing the user.

As home appliances with artificial intelligence appear, the part that interacts with people is increasing. These devices are becoming more popular as they reduce the user's trouble and make life easier. Recently, with the development of robot technology equipped with artificial intelligence, there are reports of robots that cook themselves or robots that serve them. As the number of devices that move or operate while possessing artificial intelligence around users increases, there is also a demand for more continuous monitoring and interaction.

In order to interact with mobile devices and users, devices need a function that can recognize and respond to others just like humans. With the development of IT technology, the performance of electronic devices is improving and the volume is becoming incomparably smaller. In the field of image processing, artificial intelligence functions that were previously only possible in very large and high-performance systems can now be implemented in a device with a size of a few tenths.

In the past, there were many cases in which high-performance processing devices were used for artificial intelligence image processing, or image data was transmitted over a wired or wireless network, then image processing was performed, and the results were received again through the network. However, recently, based on technological improvement, AI image processing has become possible with a small mobile device itself, and products using it have been introduced or are being prepared in real life.

An object of the present invention is to recognize a user from an optical camera image using artificial intelligence image processing S/W embedded in the device, extract the user's location, and control the movement of a camera and a mobile device that follows the user in real time. is in providing.

By responding to the user's movement in real time, it suppresses malfunctions caused by missing the user and lays the foundation for stable interaction with the user.

In order to carry out the present invention, a camera module capable of recording video is provided. The camera module continuously generates image frame data. In terms of performance, resolution, frame rate (frame per second, FPS), field of view (FOV), etc. may be considered. Since real-time video must be used, a frame rate of 5 or higher, preferably 30 or higher, is required. Additionally, the camera module may have additional lighting for use in dark environments.

In order for the camera to follow the user in the vertical direction, a mechanism for rotating the camera module in the vertical direction in a predetermined unit is provided, such as a servo motor or a step motor. The rotation angle unit has the ability to adjust the vertical angle in units of up to 1 degree, preferably 0.5 degrees or less.

A moving body capable of turning left and right and moving forward and backward requires a mechanism composed of two wheels and one caster. The wheel is driven by a power device such as a DC motor, an AC motor, a servo motor, or a step motor, but preferably a gear having a gear ratio of 1/10 or more and 1/500 or less is mounted in order to increase torque and prevent rapid movement. A plastic gear may be used as the gear, but durability and numerical accuracy can be secured by preferably using a metal gear. Preferably, a sensor capable of measuring the speed of the wheel may be attached to measure the individual speed of the wheel and adjust the speed of all wheels.

The moving body consists of a battery for supplying power, a voltage control circuit, and a charging circuit. Preferably, it includes a proximity sensor for assisting the movement of the moving object, an LED for lighting that can check the state of the moving object, and the like.

The data processing device includes a central processing unit (CPU), an image processing unit (GPU), a memory (RAM), an auxiliary memory (SD card or HDD), an input/output terminal (Video/Audio/USB, etc.). Recently, you can purchase a single board computer that has the main functions of a computer in a small area. These Single Board Computers have most of the desktop functions except for the output (display) and input (keyboard/mouse) functions, and the performance is comparable to that of a desktop computer, but the area is the size of a business card. Operating systems such as Linux-based Raspbian OS, Android OS, and Windows are installed on a Single Board Computer, and various programming languages are used to drive moving objects, control various hardware, and produce image processing programs.

The user's continuous image frames generated by the camera are extracted from the user's position data using the functional function of the vision software and the logic processing algorithm of the programming language, and the direction of the camera and the moving object is changed with the extracted data. In addition, after measuring the distance between the user and the moving object with a distance measuring device of the moving object, the moving object is moved forward and backward to maintain a certain distance from the user.

In the present invention, unlike many existing artificial intelligence devices such as artificial intelligence speakers, there is no need to exchange information through a network in the process of recognizing and tracking users, so there is no delay in information delivery. Therefore, there is no load on the network, and the mobile device does not have a constraint of connectivity to the network.

Since the device of the present invention monitors a user with a real-time image, and enables direction change and distance adjustment according to the user's motion, while minimizing restrictions on the user's activity, a separate action for using the device is not required.

The device of the present invention can be easily implemented with a smaller usage area than a business card size computer and a commercial robot vacuum cleaner, so that it does not have a large burden on space even indoors.

Because it has the above characteristics, it is very easy to implement a life-related service using the mobile body of the present invention in real life.

1 is an example of using the device
2 is a flow chart of a moving object;
3 is a flowchart of single-frame image processing and operation

The motion of the moving object of the present invention is to analyze the continuous image frame data generated by the camera to extract the user's location, use the extracted data to change the direction of the camera and the moving object, and to measure a certain distance from the user with a distance measuring device. It consists of steps to maintain.

The camera continuously generates image frames including the user and the background, and reads them in a three-dimensional color data format using an algorithm in open source vision software (OpenCV). In order to reduce the size of data in the processing device and speed up image processing, it is converted into 3D black and white data. The converted data is processed with an artificial intelligence human recognition algorithm (haar or machine learning face recognition) to distinguish the user area from the background as a region of interest. Preferably, the user's facial features are taken and the user's face part is divided into a region of interest (ROI).

The area occupied by the ROI in the camera image frame varies according to the distance between the camera and the user. As the distance between the user and the camera increases, the size of the ROI increases. Since the processing time of an image frame is proportional to the area, the smaller the area, the faster the processing is possible, but the accuracy of user recognition deteriorates instead. Therefore, in order to obtain the desired level of accuracy and processing speed, the area of the ROI is constantly reset in consideration of the specifications of the data processing device and the camera. In this case, the center of the region of interest is used as it is, and only the area of the region of interest is uniformly adjusted.

The difference between the center of the region of interest and the center of the entire image frame is calculated as an error. Most of the errors are calculated in the left-right and up-down directions of the image frame. Alternatively, the azimuth and distance may be expressed based on the center of the image frame. If the error is reduced, the camera and the moving object face the user again.

In order to reduce the left-right error, the wheels of the moving object are moved to turn left or right. The final range of rotation has a value proportional to the error. In fact, it is difficult to accurately match the driving of the motor and the movement of the screen on a one-to-one basis. To accurately match the center of the region of interest to the center of the image frame, it is possible only after many attempts. As time passes during multiple movements, real-time response becomes difficult. Therefore, a certain area is set around the center of the image frame, and when the center of the area of interest enters this area, the left and right rotation is stopped. The magnitude of the moving object movement feedback according to the error can be obtained as a proportional value through a number of trials. The moving object feedback is adjusted by setting the power or voltage delivered to the motor constant and then changing the power or voltage holding time.

In order to reduce the vertical error, if the servomotor of the camera module rotation mechanism is driven for a certain amount of time, it rotates in proportion to the driving time. The accuracy of rotation needs to be adjusted in units of 1 degree, preferably in units of 0.5 degrees, based on 360 degrees.

Through the above process, it is possible to reduce an error in a single image frame, and when the user moves, the process is repeated in successive image frames.

In a real-world use environment, the user does not always have a constant orientation with respect to the camera, but can constantly change it. At first, there is a face in the direction of the camera, but after changing the posture, it can show a side view or the back of the head. In this case, the AI face recognition algorithm cannot find the user in the subsequent image frame. Therefore, the region of interest is maintained by using a tracker algorithm that consistently tracks the movement of the region of interest in subsequent frames in which the region of interest is set by face recognition.

In order to monitor and change direction of the user in real time, image processing and direction change feedback of at least 5 frames per second, preferably 10 frames per second, and more preferably 30 frames per second are required. At a processing rate of 5 frames per second, there is a high chance of missing a sudden user movement. The 10 frames per second processing rate tracks daily movements well. The frame processing speed varies depending on the data size of the processed image frame, the artificial intelligence algorithm, and the specifications of the data processing device.

If the moving object responds to a direction change toward the user in real time through the above process, the moving object may continue to follow the user. Thereafter, the user moves forward and backward while maintaining a certain distance in order to avoid unwanted contact with the moving object in the radius of activity.

The function of measuring the distance to the user uses an ultrasonic sensor, an IR sensor, and a lidar. All measure the distance by irradiating ultrasonic waves or light in the measurement direction and detecting the ultrasonic waves or light reflected from the surface of a user or object existing in the measurement direction. In order to respond to the user's movement in real time, measurement accuracy of 10 cm or less is required, and a distance of at least 1 m must be measured in consideration of the human motion range. In addition, in order to respond to user actions in real time, the measurement processing time should be less than several hundred ms.

When the distance between the user and the moving object is measured, it moves forward or backward based on the range of 0.9 m or more and 1.5 m or less. If the distance is less than 0.9 m, it is too close to the user, so back up and spread more than 0.9 m. If the distance is more than 1.5 m, it is difficult to interact with the user, so move forward so that it is less than 1.5 m and narrow the gap.

1 is a schematic diagram of a situation in which a user and a moving object having a camera interact in the same space.

Figure 2 shows that the movement of the moving object is divided into direction change and forward/backward movement necessary for real-time monitoring.

3 shows the user is recognized (2) in the first image frame (1) for direction change, a certain region of interest is set (3), the error with the frame center is calculated (4), and the camera module is rotated up and down ( 5) and shows the sequential process of adjusting the direction toward the user by rotating the moving object left and right (6).

In a sequential process, the vertical rotation and left and right rotation are independent of the order. If the processing speed of the image frame is 10 frames per second or more, the sequential movements of the camera and the moving object are performed in about 0.1 second, so they appear to move almost simultaneously.

In order to process the image processing and control operations described above in real time on a single board computer, a parallel processing process of the program is applied. Since recent single board computers are equipped with multi-core CPUs, in order to use the operation processing of all cores, the program is divided into functional unit modules and each module runs a separate control flow (thread or process) at the same time. It bundles camera image processing and direction change into one module, and distance measurement and forward/backward movement into one module. Drives a separate handler module for data exchange between modules and movement of control flow. In this way, since the direction change and forward/backward movement data processing and feedback are performed as separate control flows, even if a bottleneck occurs in one part, the burden on the overall processing can be minimized.

Figure 3-1 Image capture frame
Figure 3-2 Frame that recognizes the user
Figure 3-3 Frame for setting a certain region of interest

Claims (10)

A camera module that generates a real-time image of the user, a camera module vertical rotation mechanism, a moving object that can change the direction left and right and forward and backward, a single board computer that stores and executes programs, artificial intelligence that distinguishes the user from the background in the image frame, and the user area Mobile device and operation method equipped with artificial intelligence and distance measuring device for setting and tracking an area of interest in an image frame In claim 1, the camera module may generate an image in units of frames, and for real-time image processing, preferably generates an image of 10 frames per second or more, more preferably 30 frames per second or more, and transmits the image to the data processing device, A module containing a light source for illumination. In Paragraph 1, the camera module vertical rotation mechanism is a power device that fixes and supports the camera module and can control quantitative rotation, such as a servo motor or a step motor, and rotates the camera in the vertical direction. Rotating mechanism that can adjust the angle in units of 1 degree or less In Paragraph 1, the moving object can move forward, backward, left turn, right turn, etc. Specifically, the wheels and the motor that drives the wheels, the gear that controls the number of rotations of the motor, the parts connecting the rotating shaft of the gear and the wheel, the rechargeable battery, etc. device comprising In Paragraph 1, Single Board Computer includes OS, is capable of analog input/output, stores device driving program, camera image processing program, etc., CPU, memory (RAM), storage medium (SD card or HDD), wired/wireless network , business card size computer board equipped with USB terminal, etc. In claim 1, the distance measuring device is a device that can measure the distance between the mobile device to be invented and the user or the mobile device and the object facing the front of the moving body to be invented by using reflection characteristics such as light and ultrasonic waves with an accuracy of 10 cm or less for a distance of 1 meter or more. Device In item 1, the artificial intelligence that distinguishes humans from the background is artificial intelligence that distinguishes users from the image frame generated by the camera, and continuously designates a specific part of the user, for example, the face, as a region of interest (ROI) with respect to a certain pixel area. Haar or face recongnition algorithm built into OpenCV as an intelligent program The AI tracking in Paragraph 1 is a tracker built into OpenCV that tracks the image of the region of interest in consideration of motion in continuous image frames, and algorithms such as meanshift, camshift, Mosse, CSRT, and KCF. As one of the programs in Paragraph 1, using the Python programming language multi-thread or multi-process, vision processing and moving object direction change in one process, moving object distance measurement and forward/backward in one process A multi-processing program that operates independently by dividing it into a handler process that manages process interest and control flow between processes The direction change of the camera module and moving object by image processing consists of repeating user image acquisition, user discrimination, ROI designation, error calculation, camera module direction change, and moving object direction change. It consists of repeating forward or backward of

Images