KR100856880B1 - Real-time image processing apparatus and method - Google Patents

Abstract

Disclosed are a real-time image processing apparatus and a method thereof. The input images are sequentially stored, the stored images are sequentially compared, and if the change between the two images is less than or equal to a predetermined threshold, the later stored images are deleted later, and the stored images are decoded. The image is previewed separately from the decoding process of the image, and the preview is stopped while the image is being stored, and the preview is executed again when the image is stored. As a result, it is possible to remove the interruption of the preview and to decode a good quality image to provide a corresponding service.

Description

Real-time image processing apparatus and method

1 is a diagram showing the configuration of an embodiment of a real-time image processing apparatus according to the present invention;

2 is a view showing an embodiment of a buffer in which an image is stored in a real-time image processing apparatus according to the present invention;

3 is a flowchart illustrating a flow of an embodiment of a real-time image processing method according to the present invention.

The present invention relates to an image processing apparatus and a method thereof, and more particularly, to a real-time image processing apparatus and method that can receive a video such as a bar code and the like in real time to provide a corresponding service.

Cameras are built into various mobile devices such as cell phones and PDAs. By using the built-in camera to recognize the objects included in the input image it is possible to apply to a variety of services. For example, face recognition, fingerprint recognition, image code recognition, logo recognition, gesture recognition and the like.

These services commonly provide a service required by a user by extracting pattern and color information included in an image from an image and recognizing or decoding the extracted information. In order to extract information from images, many processes such as noise reduction, binarization or segmentation, image enhancement, color correction, outline tracking, object extraction, feature point extraction, matching, error detection and correction are required. In particular, preprocessing techniques are used to improve an image to facilitate image recognition, to extract an object, and to prepare to find a feature point or information of the extracted object by matching. Therefore, preprocessing techniques are essential because they are important factors that determine the performance of image recognition. However, since the conventional preprocessing techniques are mostly performed on the entire image, the most computation time is required during the image recognition process.

Conventional image recognition techniques can be applied in two ways. One is when continuous real-time recognition is required and when it is not. If real-time recognition is not required, a good quality image is obtained by using equipment such as a dedicated scanner and reader, and then the recognition program is executed through a computer.

If real-time recognition is required, if you use a general-purpose device such as a mobile phone or a PDA to take images continuously and perform real-time image recognition, you can acquire a continuous image to correct the image or obtain a good image until recognition. This is a case where continuous real-time recognition is required several times per second such as searching or gesture recognition or augmented reality service.

While this continuous real-time recognition is very convenient for the user, the performance of the machinery and the efficiency of the resources are very high, and can be performed smoothly. However, current mobile computing environments have relatively low processor performance, limited memory resources, and poor camera quality compared to scanners. In addition, because of the nature of the mobile computing environment is affected by a variety of lighting and media, a large amount of computation is required. As a result, processing time is slow compared to continuous image input, resulting in delay time. In addition, when the image is generally recognized first, when previewing the camera and receiving the image into the device, that is, when storing the image, the preview is stopped and the image is stored and decoded. The preview of the image is activated again.

In general, a person recognizes a moving image when 10 images per second are presented continuously. When the decoding time is delayed, the preview is delayed, so that the image is recognized as if the camera image is cut off.

Korean Patent No. 0429703 is a method for solving this problem. This discloses a configuration to sequentially perform preview stop, image storage, preview restart, image decompression and decoding. This is a feature that decoding can be performed while the preview is in operation. Since the preview is re-executed, there is an advantage that the disconnection is relatively small.However, when the decoding time is long, waiting for a new image to be received and stored. The time can be long. In other words, even if the preview drop is small, the actual decoding time is maintained, so that the time for storing and decoding the new image is also maintained. After all, when one image is stored, new images can be stored only after decoding them, so the number of times a new image is stored for a certain time is small.

SUMMARY OF THE INVENTION The present invention provides a real-time image processing apparatus and a method for decoding an optimal image by removing meaningless images among sequentially input images, and removing a disconnection of the preview according to the decoding. have.

In order to achieve the above technical problem, an embodiment of the real-time image processing apparatus according to the present invention, the image storage unit for sequentially storing the input image; An image comparison unit comparing the stored images sequentially and deleting the stored images later if a change between the two images is less than or equal to a predetermined threshold; And an image recognition unit to decode the stored image.

In order to achieve the above technical problem, an embodiment of the real-time image processing method according to the present invention, the step of sequentially storing the input image; Comparing the stored images two by one sequentially and deleting the stored images later if the change between the two images is less than or equal to a predetermined threshold; And decoding the stored image.

As a result, it is possible to remove the interruption of the preview and to decode a good quality image to provide a corresponding service.

Hereinafter, a real time image processing apparatus and a method thereof according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing the configuration of an embodiment of a real-time image processing apparatus according to the present invention.

Referring to FIG. 1, the apparatus for real-time image processing includes an image input unit 100, a preview execution unit 110, an image storage unit 120, a format conversion unit 130, an image comparator 140, and an image recognition unit 150. It is composed of

The image input unit 100 receives an image through an image recognition device (for example, a camera, a scanner, a reader, etc.), or receives an image transmitted in a file or streaming form through a storage device or a wired or wireless communication network. For example, when the present invention is implemented in a personal portable terminal such as a mobile phone or a PDA, the image input unit 100 receives an image through a camera built in or external to the personal portable terminal or a communication network to which the personal portable terminal is connected (wired or wireless). Communication: Receives images through Bluetooth (Bluetooth, WiFi, RF, Cable, Serial, Parallel) or internal and external storage devices (disk, memory, flash memory, USB storage device).

The preview executing unit 110 displays an image received by the image input unit 100 so that the user can view it through an image display device (LCD, OLED, LED, beam projector, etc.). Conventionally, the preview is stopped when the image is stored, and the preview is restarted after the stored image is recognized or decoded. However, in the present invention, as described below, the image storing and preview re-execution and the image recognition / decoding process are separated so that the preview is not interrupted according to the decoding time. That is, preview stop-image storage-preview re-execution are continuously performed at predetermined time intervals. This can be done through timers, threading techniques, or a combination of both, with more than a few images stored per second.

The image storage unit 120 sequentially stores the images input through the image input unit 100. The image storage unit may store an image in a RAM, a flash memory, or a hard disk. When a memory such as RAM is used as a buffer for storing an image, the step of converting and storing the image data into a separate file is omitted, thereby reducing the processing time delay caused by the storing.

The format converting unit 130 converts the formats of the images stored in the image storing unit 120 into a predetermined form. Depending on the type of camera, the format of the input video can be compressed in formats such as JPG, AI, PDF, or saved in various formats such as RGB565 16 bits, YUV420_MV 12 bits, YUV420_YV 12bits, CrCb, CbCr, YUV422 16 bits. . Accordingly, the format conversion unit 130 converts the format of the images stored in the image storage unit into a format in which the image recognition unit 150 can easily recognize and decode the image. Converting and storing images in bitmap format is advantageous for image comparison and recognition / decoding.

The image comparison unit 140 compares the images stored in the image storage unit 120 with each other and deletes an image having little difference in movement between the images and a low quality image. In detail, when two or more images are stored in the image storage unit 120, the image comparison unit 140 compares two consecutive images and determines the quality of the images. The image stored in the image storage unit 120 is preferably stored in a bitmap format through the format conversion unit 130, but in some cases may be stored in a format such as JPG, TIFF, GIF, RGB 24bits, YUV, etc. It may be stored in various image formats. When an image is stored in a format other than a bitmap, the image comparison unit 140 compares the bit streams of the stored images with each other, calculates the number of bits in which the difference occurs, and if the difference is more than a predetermined number, the difference between the two images. Is judged to be large. Therefore, the format converter 130 does not necessarily convert the image into a bitmap format. Of course, when the images are converted into bitmaps and stored by the format converter 130, more various comparisons and quality evaluations are possible. If the image comparator 140 needs to modify the image or process the image for image comparison, it is preferable to make a copy of the original image as it is and use it.

The image comparator 140 may apply two methods as a method of comparing images.

One is the change in movement. In other words, the motion difference between the former image and the latter image is sensed, and if the difference is greater than a predetermined threshold, the newly input image is selected. The simplest of the various methods of detecting the motion difference is to binarize the image in black and white, and compare the pixel values (0 or 1) at the same position of each of the two binary images to count the number of pixels in which the difference occurs. To get it. This method can perform very fast operation because XOR operation of bit values of two images is needed. For example, if the binarized images of the two images are called A and B, respectively, and the binarized images are each composed of m * n size pixels, the amount of change C between the two images is expressed by Equation 1 below.

Of course, the binarization process includes some amount of computation, but it can be applied as part of the process because it is a preprocessing process that is often used in image recognition or decoding process.

Another method of comparing images is the change in color and brightness. The difference between the average brightness information of two images or the change in color is calculated to evaluate whether the change between two images is greater than a specific threshold. As the brightness value, the average of RGB color channel values may be used, or the intensity (brightness) of color models such as the Y value of YUV, the V value of HSV, and the I value of HSI may be used.

The color histogram technique is the most typical method for calculating the degree of color change, which is obtained statistically using the average and variance of color values of pixels. For example, ANOVA, chi-square test, and t-test can be applied. In the comparison of the degree of color change, an excessive amount of computation may be required by using pixels of the entire image area. Therefore, a method of using a certain area of the image, for example, a shape of a rectangle (circle, circle, triangle, etc.) centered on the center of the image, a method of reducing the image, and a method of sampling pixels of the image at regular intervals. In addition, if the method of sampling a certain number of block regions is applied, the amount of image comparison computation can be reduced.

The image comparison unit 140 compares the two images in this manner, and if the change between the two images is greater than or equal to the threshold value, the image comparison unit 140 determines that the later input image (ie, the newly input image) of the two images is meaningful. That is, the image comparison unit 140 determines that there is a change in position or a color condition in the newly input image. If the decoding of the newly input image is successful, the image comparison unit 140 deletes the previous image from the image storage unit 120. On the contrary, if the newly input image does not have a big difference in movement with the previous image or there is no big difference in color, brightness, etc., the old image is left as it is and the new input image is deleted from the image storage unit 120. In addition, even if the newly input image is a meaningful image, if the decoding fails, the image comparison unit 140 maintains the previous image and deletes the newly input image from the image storage unit 120.

The image comparator 140 also determines the quality of the image. As a criterion of the image quality, whether or not an object of a specific shape can be discriminated from the corresponding image. For example, when an image code is input, an image code can be recognized only when a black pixel area obtained through binarization exists in a predetermined ratio or more. In addition, it may be determined whether the image having the information capable of recognizing and decoding the image is determined based on the shape, size, color configuration, brightness of the image, and the like. In addition, the image quality may be determined by whether the color and brightness of the image are recognizable.

The image recognition unit 150 recognizes or decodes the images stored in the image storage unit and then provides a predetermined service corresponding to the image. For example, when the image is a one-dimensional or two-dimensional barcode, the image recognition unit 150 recognizes and decodes the image to grasp the information of the barcode, and then provides a predetermined service according to the identified information.

When decoding is successful in the image recognition unit 150, a corresponding application is driven. At this time, it is necessary to determine whether to continue or stop the video input. For example, augmented reality or mixed reality services that need to continuously overlay visual information such as letters, symbols, and 3D objects on the camera preview image by recognizing image codes or gestures in the camera image. In this case, preview and image input and recognition should continue at least 10 times per second until the user artificially stops. On the contrary, if the mobile phone recognizes the code image and accesses the web site according to the code image information, or retrieves the information from the database and provides it to the user, preview and continuous video input are not required. It is preferable to end.

In general, when real time decoding is required, it is efficient to determine whether the input video is worth decoding and to decode only the worth video. Conventionally, in order to prevent the preview from being interrupted, especially when an image is input through a camera, a time difference occurs between the image currently viewed as the preview and the image to be actually decoded, and thus the decoding of the input image is delayed. There was a problem. Therefore, in the present invention, the stop of the preview, the storage of the image, and the execution of the preview are continuously performed as one process, and the comparison between images and decoding / recognition are performed in parallel as another process.

Therefore, the real-time image processing apparatus according to the present invention does not decode the image every time the image is stored, but stores the image and decoding thereof separately. For example, assume that the decoding time is 0.3 seconds at maximum, the image storage time is 0.1 seconds, and the image comparison work time is 0.1 seconds. According to the conventional method, in an environment capable of receiving 10 images per second, the storage and decoding time is 0.4 seconds in the worst case. Therefore, it is possible to store and decode 2.5 times per second regardless of the quality of the image. However, according to the present invention, since 10 images can be stored per second, and image comparison is simultaneously performed for images stored at the same time, the comparison operation of images is performed for up to 9-10 times. Therefore, the present invention creates an opportunity to decode a better quality valuable image within that time.

2 is a diagram illustrating an embodiment of a buffer in which an image is stored in a real-time image processing apparatus according to the present invention.

When there is a lot of change between input images and the decoding speed is slow, the image storage speed cannot keep up with the decoding speed and the images continue to accumulate in the buffer. In this case, there is a possibility that a picture much earlier than the current preview picture is being decoded. Therefore, in the present invention, the circular buffer 200 shown in FIG. 2 is used as the image storage unit 120 of FIG. 1.

Referring to FIG. 2, an input image is stored in a circular buffer 200 composed of a predetermined number of buffers. Therefore, when the buffer is full, the first input image is removed from the buffer and the newly input image is stored in its place. That is, the image n + 1 is stored where the image 1 is located.

3 is a flowchart illustrating a flow of an embodiment of a real-time image processing method according to the present invention.

Referring to FIG. 3, the real-time image processing method according to the present invention includes two processes in parallel. First, when an image is input (S300), the real-time image processing apparatus stops the preview (S305) and stores the image (S310). The real-time image processing apparatus re-executes the preview immediately after the image storage is completed (S315). That is, the preview is only paused when the image is stored, but does not occur due to the recognition or decoding process of the image.

Images are sequentially stored in the image storage unit 120 by this process. In this case, the image storage unit 120 may be configured of the circular buffer 200 described with reference to FIG. 2. Since the recognition / decoding process for the image is performed based on the image stored in the image storage unit 120, the interruption of the preview due to decoding does not occur.

In detail, the apparatus for real-time image processing sequentially compares two images stored in the image storage unit 120 to determine whether a change in movement, a color, and a brightness are greater than or equal to a predetermined threshold (S350). If not (S355) and after the input image is deleted from the image storage unit 120 (S360). If there is a change over the threshold, the real-time image input apparatus performs a recognition / decoding process for the newly input image (S360), and provides a corresponding service according to the result. If decoding fails, the newly input image is deleted and the previous image is kept as it is.

The invention can also be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet). Include. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

According to the present invention, there is no interruption of the preview screen in image recognition and decoding through the camera interface using the preview. In addition, since image storage and decoding are not performed sequentially but are performed in separate parallel processing, it is possible to examine more image quality and select and decode meaningful good quality images within the same time. Performance is improved. In addition, even if the image is already stored or provided in real time through a network, video, etc., it is possible to find, recognize, and decode the image of meaningful quality.

Claims (15)

An image storage unit sequentially storing the input image; An image comparison unit comparing the stored images sequentially and deleting the stored images later if a change between the two images is less than or equal to a predetermined threshold; And And an image recognizing unit to decode the stored image. The method of claim 1, And a preview execution unit which previews the input image and stops the preview while the image is stored in the image storage unit and then re-executes the preview when the image storage is completed. . The method of claim 1, And a format converting unit converting the images stored in the image storing unit into a bitmap format. The method of claim 1, And the image storage unit cyclically stores the input image in a circular buffer having a predetermined number. The method of claim 1, The image comparator binarizes the two images, and deletes the later stored image if the movement change between the two images determined based on the difference between the pixel values of the same position of the binarized images is less than or equal to a predetermined threshold. Processing unit. The method of claim 1, The image comparator compares brightness between two images based on an average of RGB color channel values, Y value of YUV, V value of HSV, or I value of HSI, or color change between two images obtained through a color histogram method. And a later stored image is deleted later than a predetermined threshold value. The method of claim 1, And the image comparison unit determines whether the stored image can be recognized using color, brightness, and object size information of the image, and deletes the image if image recognition is impossible. Sequentially storing the input images; Comparing the stored images two by one sequentially and deleting the stored images later if the change between the two images is less than or equal to a predetermined threshold; And Decoding the stored image; real-time image processing method comprising a. The method of claim 8, Previewing the input image; Stopping the preview while the input image is stored; And And re-executing the preview when the storing of the input video is completed. The method of claim 8, And converting the stored images into a bitmap format. The method of claim 8, The image storing step includes the step of circularly storing the input image in a circular buffer consisting of a predetermined number; real-time image processing method comprising a. The method of claim 8, The deleting of the image may include binarizing the two images, and deleting the stored image later if a change in motion between the two images based on a difference between pixel values of the same position of the binarized images is less than or equal to a predetermined threshold. Real-time image processing method characterized in that. The method of claim 8, The image erasing step may be performed by changing the brightness between two images based on the average of RGB color channel values, the Y value of YUV, the V value of HSV, or the I value of HSI, or the color change between two images identified through a color histogram. And deleting the later stored image when the value is equal to or less than a predetermined threshold. The method of claim 8, Deleting the image Determining whether the stored image can be recognized using the color, brightness, and object size information of the image, and if the image is not recognized, deleting the image; Treatment method. A computer-readable recording medium having recorded thereon a program for executing a real-time image processing method according to any one of claims 8 to 14 on a computer.

Images