KR102599584B1 - Image processing apparatus - Google Patents

Abstract

An image processing device according to an embodiment of the present invention includes an image input unit for inputting image data captured by a camera, an object extraction unit for extracting at least one object included in the image data, and at least one of the at least one object. It includes an object selection input unit that receives an input for selecting, a frame reconstruction unit that reconstructs a plurality of frames included in the image data centered on the selected object, and a rendering unit that renders using the reconstructed frames.

Description

Image processing device {IMAGE PROCESSING APPARATUS}

The present invention relates to an image processing device that can generate a shake-free image for a selected subject from an image in which shaking occurs.

In general, when the camera that captures the video shakes, such as when taking video in a car driving on a bumpy road or taking video while doing extreme sports, the video shakes, making it difficult to observe the subject, and causing considerable fatigue during observation. There is a problem that arises.

Conventionally, an OIS (Optical Image Stabilization) actuator was used to directly move the lens in the direction corresponding to the camera shake to capture a shake-free image, or a shake vector was extracted by analyzing the shaken image using DIS (Digital Image Stabilization). , using this, a shake-free image was created using a software correction method for the shaken image.

However, when using an OIS actuator, there is a limit to how much the camera lens can be moved, so if shaking occurs more than the above limit, there is a problem in that it cannot be corrected. In the case of DIS, if the amount of shaking is large, the motion vector estimation method This has the disadvantage of being difficult and requiring a lot of computation due to the complex algorithm.

Korean Patent Publication No. 10-1467869

The present invention is intended to solve the problems of the prior art described above. By reconstructing and rendering a frame centered on an object selected by the user among the objects included in the image data, it is possible to generate a shake-free image for the object requiring observation. Provides an image processing device that can

An image processing device according to an embodiment of the present invention includes an image input unit for inputting image data captured by a camera, an object extraction unit for extracting at least one object included in the image data, and at least one of the at least one object. An object selection input unit that receives an input for selecting, an object detection unit that detects the selected object in a plurality of frames included in the image data, moves the position of the plurality of frames around the detected object, and the position is It includes a frame reconstruction unit that synthesizes a plurality of moved frames to generate a composite image, and a rendering unit that renders using the synthesized frames.

In one embodiment, the frame reconstruction unit may change the angle of view and position of the frame to place the selected object in the center of the screen.

In one embodiment, when one of the at least one object is selected through the object selection input unit, the display may further include an object information display unit that extracts and displays information about a creation or extinction time for the selected object.

In one embodiment, when there are a plurality of objects extracted by the object extraction unit and a plurality of objects are selected by the object selection input unit, the frame reconstruction unit, for each of the selected plurality of objects, includes a plurality of objects included in the image data. A plurality of frames can be reconstructed, and the rendering unit can divide the screen according to the number of objects and render the plurality of frames.

Here, the frame reconstruction unit may reconstruct the plurality of frames so that the selected object is located in the center of each divided frame compared to other objects.

According to one embodiment of the present invention, there is an effect of generating a shake-free image for an object requiring observation by reconstructing and rendering a frame centered on an object selected by the user among objects included in image data.

1 is a configuration diagram for explaining an image processing device according to an embodiment of the present invention.
Figure 2 is a configuration diagram for explaining an image processing device according to another embodiment of the present invention.
Figure 3 is a diagram for explaining a plurality of frames included in video data.
FIG. 4 is a diagram for explaining an embodiment of the operation of the frame reconstruction unit of FIG. 1.
FIG. 5 is a diagram for explaining an embodiment of the object information display unit of FIG. 2.
Figure 6 is a diagram for explaining a case where there are multiple objects.
Figure 7 is a diagram for explaining an example of the operation of the rendering unit when a plurality of objects are selected.

Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings.

However, the embodiments of the present invention may be modified into various other forms, and the scope of the present invention is not limited to the embodiments described below. Additionally, the embodiments of the present invention are provided to more completely explain the present invention to those with average knowledge in the relevant technical field.

In the drawings referenced in the present invention, components having substantially the same configuration and function will be given the same symbols, and the shapes and sizes of elements in the drawings may be exaggerated for clearer description.

FIG. 1 is a configuration diagram for explaining an image processing device according to an embodiment of the present invention, and FIG. 2 is a configuration diagram for explaining an image processing device according to another embodiment of the present invention.

Referring to FIG. 1, the image processing device 20 according to an embodiment of the present invention includes an image input unit 100, an object extraction unit 200, an object selection input unit 300, an object detection unit 400, and a frame reconstruction unit ( 500) and a rendering unit 600.

In one embodiment, the image processing device 20 may further include an object information display unit 700, as shown in FIG. 2 .

Image data captured by the camera 10 may be input to the image input unit 100 . Here, the image input unit 100 may transmit the image data to the object extractor 200. The image input unit 100 may be composed of an image acquisition device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) that receives an image signal from a subject. Additionally, the image signal may be a digital signal expressed in three color components, such as RGB and YUV.

The object extractor 200 may analyze image data transmitted from the image input unit 100 and extract at least one object included in the image data. In one embodiment, the object extractor 200 may be set to extract only objects that satisfy preset conditions among objects included in image data. Here, at least one extracted object may be displayed on the screen for the user to select. The object extractor 200 can extract feature information for each object using a single image. For example, it is possible to perform the characteristic information extraction function included in the object detection unit 130 on a separate chip. The functions performed by the object extraction unit 100 are preferably performed using a single image rather than a combined image.

The object selection input unit 300 may receive an input for selecting at least one object among at least one object extracted by the object extraction unit 200. Here, the object selection input unit 300 may be implemented by including various input means that can receive input for selecting at least one of a plurality of objects, such as a keypad, mouse, touchpad, etc. The object selection input unit 300 may transmit information about the selected object to the frame reconstruction unit 500.

The object detection unit 400 may detect the object selected by the object selection input unit 300 from each of a plurality of frames included in the image data input through the image input unit 100. Here, the object detection unit 400 can detect the object extracted by the object extraction unit 100 by tracking its location. Here, the object detector 400 may track the location of the object using a pattern matching algorithm or an optical detector to reduce the amount of calculation. Additionally, in order to remove cumulative errors occurring during the tracking process, the object detector 400 may correct location information using a single image at each preset period.

The frame reconstruction unit 500 moves the positions of a plurality of frames included in the image data around the object detected by the object detection unit 400, and synthesizes the plurality of frames whose positions have been moved to generate a composite image. You can. In one embodiment, the frame reconstruction unit 500 may change the angle of view and position of each of the plurality of frames so that the selected object is fixed to a preset coordinate (e.g., the center of the screen) on the screen to display the image data. there is. Accordingly, the synthesized image may become a partial area within the entire screen area, and as the number of synthesized images increases, the visible area may gradually become wider in the entire screen area. Here, the number of synthesized images may be defined as a predetermined time interval or the number of frames.

In addition, when a plurality of objects exist in the composite image, and one object is selected by the object selection input unit 300, the frame reconstruction unit 500 displays the remaining objects in shading or blinking. can do.

In addition, the frame reconstruction unit 500 can change the center of the screen around the other object when a plurality of objects exist in the composite image and select one object among them and then select another object. there is.

In addition, the frame reconstruction unit 500 stores the first frame in which the object appears as an I-frame, and frames after the I-frame contain only information about the selected object or information about the newly appearing object. It can be saved as a P-frame, and objects other than the selected object or newly appearing object can be processed in a single color.

In addition, when a plurality of objects are selected by the object selection input unit 300, the frame reconstruction unit 500 divides the screen according to the number of selected objects, reconstructs a frame for each of the selected plurality of objects, and It can be placed at the appropriate location.

The rendering unit 600 may render the synthesized image based on the frame synthesized by the frame reconstruction unit 500 and reproduce the synthesized image.

FIG. 3 is a diagram for explaining a plurality of frames included in video data, and FIG. 4 is a diagram for explaining an embodiment of the operation of the frame reconstruction unit of FIG. 1.

In one embodiment, if the camera 10 is shaken during filming, the image captured while the camera 10 is shaken is shaken, making it difficult to observe a subject that needs to be observed, and causing significant fatigue.

The present invention is intended to solve this problem. When a user selects an object that needs to be observed, the frame is reconstructed around the object, thereby creating a shake-free image for the selected object.

In one embodiment, when the camera 10 is shaken during shooting, as shown in FIG. 3, a plurality of frames included in the image data input through the video input unit 100 produce images that are shaken up and down or left and right. It can be included.

In this case, the object extractor 200 may extract the first object 1, the second object 2, and the third object 3. Here, when the first object 1 is selected through the object selection input unit 300, the frame reconstruction unit 500 moves the first object 1 to a preset position (d) on the screen d, as shown in FIG. 4. For example, the angle of view and position of the first frame (a), the second frame (b), and the third frame (c) can be changed so that they are in the exact center of the screen (d). Here, the rendering unit 600 may generate a shake-free image for the first object 1 by rendering according to the reconstructed frame.

Here, the synthesized image (frames a, b, c) may be a partial area within the entire screen area (d), and as the number of synthesized images increases, the visible area may gradually become wider in the entire screen area. Here, the number of synthesized images may be defined as a predetermined time interval or the number of frames.

In addition, when a plurality of objects (1, 2, and 3) exist in the composite image, and one object (1) is selected by the object selection input unit 300, the frame reconstruction unit 500 selects the remaining objects. (2, 3) can be displayed in shading or blinking.

FIG. 5 is a diagram for explaining an embodiment of the object information display unit of FIG. 2.

Referring to FIG. 5 , the object information display unit 300 may display information about an object selected through the object selection input unit 300. Here, the information about the object may include, for example, information about the time, date, and location when the object occurred and the time, date, and location when the object disappeared.

In one embodiment, when the first object 1 is selected through the object selection input unit 300, the frame reconstruction unit 500 reconstructs the frame centered on the first object 1, but uses an I-frame (FIG. 5 If an object 2 that is not included in the first frame (a)) is found in a subsequent frame (second frame (b) in FIG. 5), the object 2 may be classified as an object of interest, The rendering unit 600 may highlight the newly discovered object of interest 2 or display the area of the object of interest 2 by blinking.

FIG. 6 is a diagram for explaining a case where there are multiple objects, and FIG. 7 is a diagram for explaining an embodiment of the operation of the rendering unit when multiple objects are selected.

Referring to FIG. 6, image data input through the image input unit 100 may include a plurality of objects (1, 2, 3). Here, the frame reconstruction unit 500 reconstructs the frame around the object selected through the object selection input unit 300, and when a plurality of objects are selected, it reconstructs the frame for each of the selected objects, as shown in FIG. 7. This can be done, and the rendering unit 600 can divide the screen d according to the number of the selected objects 1, 2, and 3 and generate an image using the reconstructed frame for each object. For example, in the first split screen d1, the frame can be reorganized so that the first object 1 is located at the center of the first split screen d1. Likewise, the frame can be reorganized so that the second object 2 is located at the center of the second split screen d2, and the frame can be reorganized so that the third object 3 is located at the center of the third split screen d3. It can be reconstructed.

Here, although not shown, the object information display unit 700 can display information about the first object 1 on the first split screen d1 that plays the image about the first object 1, and The second split screen (d2) that plays the image for the second object (2) can display information about the second object (2), and the third split screen that plays the image for the third object (3) In (d3), information about the third object 3 can be displayed.

The present invention described above is not limited by the above-described embodiments and the accompanying drawings, but is limited by the scope of the patent claims described below, and the configuration of the present invention can be varied within the scope without departing from the technical spirit of the present invention. Those skilled in the art can easily see that changes and modifications can be made.

10: Camera
20: Image processing device
100: Video input unit
200: Object extraction unit
300: Object selection input unit
400: frame reconstruction unit
500: Rendering unit
600: Object information display unit

Claims (5)

An image input unit where image data captured by a camera is input;
an object extraction unit that extracts at least one object included in the image data;
an object selection input unit that receives an input for selecting at least one object that needs to be observed among the at least one object;
an object detector detecting the selected object among at least one object included in the plurality of frames, from a plurality of frames included in the image data;
a frame reconstruction unit that moves the positions of the plurality of frames around the detected object and synthesizes the plurality of frames whose positions have been moved to generate a composite image; and
a rendering unit that renders using the synthesized frame;
Including,
The frame reconstruction unit is configured to change the angle of view and position of the plurality of frames so that the detected object is maintained at a preset position on the screen. delete According to paragraph 1,
an object information display unit that, when one of the at least one object is selected through the object selection input unit, extracts and displays information about a creation or destruction time of the selected object; An image processing device further comprising: According to paragraph 1,
When there are a plurality of objects extracted from the object extraction unit and a plurality of objects are selected by the object selection input unit,
The frame reconstruction unit reconstructs a plurality of frames included in the image data for each of the plurality of selected objects,
The rendering unit divides the screen according to the number of objects and renders the plurality of frames. The method of claim 4, wherein the frame reconstruction unit,
An image processing device that reconstructs the plurality of frames so that the selected object is located in the center of each divided frame compared to other objects.

Images