KR20230068979A - Imaging device - Google Patents

Abstract

An imaging device according to one aspect of the technical idea of the present disclosure includes an image sensor which generates a plurality of original images and an image processor which performs an operation to generate a merged image based on the plurality of original images. The image processor classifies the plurality of original images into a plurality of stitching groups based on the first stitching direction, generates a plurality of intermediate images by stitching original images included in the same stitching group, and generates a merged image by stitching the plurality of intermediate images based on the second stitching direction.

Description

Imaging device {IMAGING DEVICE}

The technical idea of the present disclosure relates to an imaging device, which stitches a plurality of original images.

The electronic device may provide a camera function capable of generating and storing an image or video by photographing an external object. In this case, in order to obtain an image having a wide angle of view or an image having high resolution, the electronic device may stitch a plurality of images generated through an image sensor.

In this case, the plurality of images may be arranged in both directions (eg, horizontal and vertical directions) instead of in one direction (eg, horizontal direction). Therefore, it is necessary to develop a method of stitching images disposed in various directions.

In addition, when a plurality of images are stitched together to generate one image, a large amount of computation and a large amount of processing time are required, so it is necessary to develop a method to reduce this.

An object to be solved by the technical idea of the present disclosure is to provide an imaging device capable of stitching images disposed in various directions.

In addition, an object to be solved by the technical concept of the present disclosure is to provide an imaging device capable of stitching images with a small amount of calculation and a small processing time.

In order to achieve the above object, an imaging device according to an aspect of the present disclosure performs an image sensor for generating a plurality of original images and an operation for generating a merged image based on the plurality of original images. An image processor that classifies the plurality of original images into a plurality of stitching groups based on a first stitching direction, and stitches the original images included in the same stitching group to obtain a plurality of intermediate images. Images are generated, and the merged image is generated by stitching the plurality of intermediate images based on a second stitching direction.

In order to achieve the above object, an imaging device for generating a merged image based on a plurality of original images according to an aspect of the technical concept of the present disclosure includes a transformation matrix used for warping of the original image and the A memory for storing a cropping area representing an unnecessary area in an original image, generating the original image, generating a warped image by warping the original image using the transformation matrix, and generating the warped image by the cropping An image sensor generating a cropped image by cropping using an area, and an image processor receiving a plurality of cropped images from the image sensor and blending the plurality of cropped images to generate a merged image. do.

In order to achieve the above object, an imaging device according to one aspect of the technical idea of the present disclosure stores a transformation matrix used for warping of an original image and a cropping region representing an unnecessary region in the original image An optical device that rotates according to rotation information including a memory, a direction and an angle, a controller that transmits a rotation signal to the optical device and generates a photographing signal corresponding to the rotation signal, and a photographing signal received from the controller. In response, a target object is photographed to generate an original image, the original image is warped using the transformation matrix to generate a warped image, and the warped image is cropped using the cropping area to crop It includes an image sensor that generates an image and an application processor (AP) that receives a plurality of cropped images from the image sensor and generates a merged image by blending the plurality of cropped images.

According to the imaging device of the technical concept of the present disclosure, since a merged image is generated by stitching a plurality of original images based on first and second stitching directions, images disposed in various directions may be stitched.

In addition, according to the imaging device of the technical concept of the present disclosure, since a plurality of original images are stitched based on a transformation matrix calculated in an overlapping area and a seam detected in the overlapping area, a merged image can be obtained with a small amount of calculation and a small processing time. can create

Also, according to the imaging device of the technical concept of the present disclosure, since an image sensor warps and crops an original image, and an image processor blends a plurality of cropped images, a merged image can be generated with less processing time.

1 is a block diagram illustrating an imaging device according to an exemplary embodiment of the present disclosure.
2 is a diagram illustrating an optical device included in an imaging device according to an exemplary embodiment of the present disclosure.
3 is a diagram illustrating a plurality of original images generated through an imaging device according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method of generating a merged image by an imaging device according to an exemplary embodiment.
5A to 5D are diagrams illustrating a process of generating a merged image from a plurality of original images by an imaging device according to an embodiment of the present disclosure.
6 is a flowchart illustrating a method of stitching a plurality of original images by an imaging device according to an embodiment of the present disclosure.
7 is a flowchart illustrating a method of calculating a transformation matrix between a plurality of original images by an imaging device according to an embodiment of the present disclosure.
8 is a flowchart illustrating a method of detecting a seam between a plurality of original images by an imaging device according to an embodiment of the present disclosure.
9 is a flowchart illustrating a method of blending a plurality of original images by an imaging device according to an embodiment of the present disclosure.
10 is a block diagram illustrating an imaging device according to another exemplary embodiment of the present disclosure.
11A to 11D are diagrams illustrating a process of generating a merged image from a plurality of original images by an imaging device according to another embodiment of the present disclosure.
12 is a flowchart illustrating a process of generating a merged image from a plurality of original images by an imaging device according to another embodiment of the present disclosure.
13 is a flowchart illustrating a process of generating an original image by an imaging device according to another embodiment of the present disclosure.
14 is a flowchart illustrating a process of generating a merged image from a plurality of original images arranged in both directions by an imaging device according to another embodiment of the present disclosure.
15 is a diagram illustrating a process of generating a cropped image by cropping in one direction by an imaging device according to another embodiment of the present disclosure.
16 is a block diagram illustrating an imaging device according to another exemplary embodiment of the present disclosure.
17A and 17B are block diagrams illustrating an imaging device including a plurality of imaging devices according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an imaging device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 1 , an imaging device 100 according to an embodiment of the present disclosure may include an image sensor 110 , an image processor 120 , a memory 130 and an optical device 140 .

The imaging device 100 may capture and/or store an image of a subject by using a solid-state image sensor such as a charge-coupled device and a metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS), and may include a digital camera, a digital camcorder, and a mobile phone. , or a tablet computer, or may be implemented as part of a portable electronic device. Portable electronic devices include laptop computers, mobile phones, smartphones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital still cameras, digital video cameras, audio devices, portable multimedia players (PMPs), and PNDs. (Personal Navigation Device), MP3 player, handheld game console, e-book, wearable device, and the like. In addition, the imaging device 100 may be used in electronic devices such as drones and advanced driver assistance systems (ADAS), or electronic devices provided as parts of vehicles, furniture, manufacturing facilities, doors, and various measuring devices. can be mounted.

The image sensor 110 may generate a plurality of original images. A plurality of original images are images with different views of a scene, and the image sensor 110 captures a target object with different views through an optical device 140 rotatable in a preset direction and angle. A plurality of original images can be created. The image sensor 110 may output a plurality of generated original images to the image processor 120 . The image sensor 110 may output a plurality of indices corresponding to each of a plurality of original images to the image processor 120 . The index may represent spatial coordinates of the original image. The image sensor 110 may be mounted in an electronic device having an image or light sensing function.

The image processor 120 may receive a plurality of original images from the image sensor 110 . Also, the image processor 120 may perform image processing on a plurality of original images.

In one embodiment of the present disclosure, the image processor 120 may perform an operation for generating a merged image based on a plurality of original images. The image processor 120 classifies a plurality of original images into a plurality of stitching groups based on a first stitching direction, creates a plurality of intermediate images by stitching the original images included in the same stitching group, and A merged image may be generated by stitching a plurality of intermediate images based on two stitching directions.

At this time, the image processor 120 blends the original images included in the same stitching group based on the transformation matrix calculated in the overlapping area between the original images included in the same stitching group and the seam detected in the overlapping area. In this way, original images included in the same stitching group may be stitched. Also, the image processor 120 may stitch the plurality of intermediate images by blending the plurality of intermediate images based on the transformation matrix calculated in the overlapping region between the plurality of intermediate images and the seam detected in the overlapping region.

1 shows an embodiment in which the image processor 120 is configured separately from the image sensor 110 in the imaging device 100, but the present disclosure is not limited thereto, and a part of the image processor 120 may include the image sensor 110 ), or may be located inside a separate processor outside the imaging device 100.

The memory 130 may store data required for image processing. The memory 130 may be implemented as volatile memory or non-volatile memory. Volatile memory may include DRAM (Dynamic Random Access Memory), SRAM (Static RAM), etc. Non-volatile memory may include ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Electrically Programmable ROM), EEPROM (Electrically Programmable ROM) Erasable and programmable ROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), and the like. In one embodiment of the present disclosure, the memory 130 may be a One Time Programmable (OTP) memory included in the imaging device 100 .

In one embodiment of the present disclosure, the memory 130 may store a transformation matrix used when the image processor 120 creates a merged image by stitching a plurality of original images. Also, the memory 130 may be optimized through a cropping operation of the image processor 120 or the like.

The optical device 140 may adjust the field of view of a plurality of original images by rotating according to rotation information including directions and angles. For example, the optical device 140 rotates in at least one of the x-axis, y-axis, and z-axis directions so that the image sensor 110 generates a plurality of original images having different fields of view for one scene. can do. In one embodiment of the present disclosure, the optical device 140 may acquire different scenes by moving up and down, left and right through rotation in pitch, yaw, and roll directions. The optical device 140 may rotate in a preset direction and angle based on a command received from the image processor 120 .

The optical device 140 may be an optical concentrating device including a mirror or a lens. A more detailed structure of the optical device 140 will be described later with reference to FIG. 2 .

As such, since the imaging device 100 according to the present disclosure generates a merged image by stitching a plurality of original images based on the first and second stitching directions, images disposed in various directions may be stitched. In addition, since a plurality of original images are stitched based on the transformation matrix calculated in the overlapping area and the seam detected in the overlapping area, a merged image can be generated with a small amount of calculation and a small processing time.

2 is a diagram illustrating an optical device included in an imaging device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 2 , an appearance of a prism lens of one embodiment of the optical device 140 may be confirmed. 2 illustrates a prism lens as one embodiment of the optical device 140, but the present disclosure is not limited thereto, and in another embodiment of the present disclosure, light is reflected by an object using optical characteristics such as dispersion or refraction. An optical device capable of condensing various paths of light or changing a moving path of light may be used.

The optical device 140 may include a total of three spatial rotation axes, x-axis, y-axis, and z-axis. The optical device 140 rotates around at least one of the x-axis, y-axis, and z-axis, so that the image sensor 110 generates a plurality of original images having different views of one scene. .

Rotation about the x-axis of the optics 140 may be referred to as rolling (transverse rocking). Rotation of the optics 140 around the y-axis may be referred to as pitching (vertical shake). Rotation of the optical device 140 around the z-axis may be referred to as yawing.

In one embodiment of the present disclosure, rotation in the pitch direction can be understood as rotation in the y-axis direction transversely penetrating the optical device 140, and rotation in the yaw direction is the rotation of the optical device 140. It can be understood as rotation in the z-axis direction vertically penetrating , and rotation in the roll direction can be understood as rotation in the x-axis direction penetrating the optical device 140 in the longitudinal direction.

The image sensor 110 may have a critical angle of view for capturing an image of an object. The critical angle of view may be limited by user settings or may be limited by physical limitations of the image sensor 110 . For example, when the image sensor 110 is embedded in the imaging device 100, the critical angle of view of the image sensor 110 cannot exceed 180 degrees.

In an embodiment of the present disclosure, the optical device 140 may cause the image sensor 110 to generate an original image representing a foreground subject using pitching and yawing. The optical device 140 may cause the image sensor 110 to generate an original image representing the left and right of the subject through yawing. Further, the optical device 140 may cause the image sensor 110 to generate an original image representing the top and bottom of the subject through pitching. That is, the optical device 140 may cause the image sensor 110 to generate an original image representing the top, bottom, left, and right peripheries of the subject through pitching and yawing.

3 is a diagram illustrating a plurality of original images generated through an imaging device according to an embodiment of the present disclosure.

Referring to FIG. 3 , a plurality of original images generated by the image sensor 110 may be identified. In the embodiment of FIG. 3 , the plurality of original images may include first to ninth original images, which are a total of nine original images.

Here, the number of original images generated by the image sensor 110 may be set based on the shooting mode received from the image processor 120 . For example, when the shooting mode received from the image processor 120 is the first shooting mode for generating a total of 9 original images, 3 horizontally and 3 vertically, the image sensor 110 is shown in FIG. 3 . First to ninth original images as described above may be generated. In this case, the image processor 120 may arrange the plurality of original images as shown in FIG. 3 based on the plurality of indices received together with the plurality of original images from the image sensor 110 .

As another example, when the shooting mode received from the image processor 120 is the second shooting mode for generating a total of 20 original images, 5 horizontally and 4 vertically, the image sensor 110 is configured as shown in FIG. 3 . Unlike the bar, first to twentieth original images may be generated.

Hereinafter, an embodiment of the present disclosure will be described focusing on an embodiment in which the imaging device 100 operates in the first photographing mode.

The image sensor 110 may output a plurality of indices corresponding to each of the plurality of original images to the image processor 120 along with the plurality of original images. A plurality of indices represent spatial coordinates of a plurality of original images, and the plurality of indices may be as indicated by circular numbers in FIG. 3 .

In the embodiment of FIG. 3 , the first original image may be an original image located at the upper left of the plurality of original images, have an index of 1, and spatial coordinates of (1, 1). The second original image may be an original image located at the top center among a plurality of original images, have an index of 2, and spatial coordinates of (1, 2). The third original image may be an original image located at the upper right of the plurality of original images, have an index of 3, and spatial coordinates of (1, 3). The fourth original image may be an original image located at the middle left of the plurality of original images, have an index of 4, and spatial coordinates of (2, 1). The fifth original image is an original image located in the middle of a plurality of original images, has an index of 5, and spatial coordinates of (2, 2). The sixth original image is an original image located at the right middle of the plurality of original images, has an index of 6, and spatial coordinates of (2, 3). The seventh original image is an original image located at the bottom left of the plurality of original images, and may have an index of 7 and spatial coordinates of (3, 1). The eighth original image is an original image located at the lower center of the plurality of original images, has an index of 8, and spatial coordinates of (3, 2). The ninth original image is an original image located at the lower right of the plurality of original images, has an index of 9, and spatial coordinates of (3, 3).

The optical device 140 may rotate in a preset direction and angle based on a command received from the image processor 120 . Accordingly, a plurality of original images generated by the image sensor 110 in the same photographing mode may include common geometric characteristic information. For example, the first original image created by being photographed in the first photographing mode may always rotate at the same angle as the second original image created by being photographed in the first photographing mode. Also, the first original image created by capturing in the first capturing mode may always be enlarged or reduced by the same magnification as the second original image generated by capturing in the first capturing mode.

4 is a flowchart illustrating a method of generating a merged image by an imaging device according to an exemplary embodiment.

Referring to FIG. 4 , in step S410, the image processor 120 may determine first and second stitching directions.

The stitching direction is a direction indicating in what order the original images are stitched when the image processor 120 generates a merged image based on a plurality of original images.

In one embodiment of the present disclosure, the first and second stitching directions may be a row direction or a column direction. In this case, if the first stitching direction is a row direction, the second stitching direction may be a column direction. Conversely, if the first stitching direction is a column direction, the second stitching direction may be a row direction.

For example, when the plurality of original images include the first to ninth original images as shown in FIG. 3 and the first stitching direction is in the row direction, the image processor 120 selects the original images located in the same row. You can stitch first.

In one embodiment of the present disclosure, the image processor 120 may determine first and second stitching directions based on rotation information of the optical device 140 . In this case, the image processor 120 may set the first stitching direction to be the same as the initial rotation direction of the optical device 140 . For example, when the optical device 140 first rotates in the row direction, the image processor 120 may set the first stitching direction to the row direction. Accordingly, since the image processor 120 can perform stitching in the order in which a plurality of original images are generated through the image sensor 110, from the time of generating the first original image of the imaging device 100 to the time of generating the merged image, The time interval can be minimized.

In operation S420, the image processor 120 may classify a plurality of original images into a plurality of stitching groups based on the first stitching direction.

If the first stitching direction is a row direction, the image processor 120 may classify original images located in the same row into the same stitching group. Conversely, if the first stitching direction is a column direction, the image processor 120 may classify original images located in the same column into the same stitching group.

In one embodiment of the present disclosure, when the plurality of original images include the first to ninth original images as shown in FIG. 3 and the first stitching direction is the row direction, the image processor 120 performs the first to ninth original images. The third original image may be classified as a first stitching group, the fourth to sixth original images may be classified as a second stitching group, and the seventh to ninth original images may be classified as a third stitching group.

In another embodiment of the present disclosure, when the plurality of original images include the first to ninth original images as shown in FIG. 3 and the first stitching direction is a column direction, the image processor 120 performs the first, The fourth and seventh original images are classified into a first stitching group, the second, fifth, and eighth original images are classified into a second stitching group, and the third, sixth, and ninth original images are classified into a third stitching group. can be classified as

In step S430, the image processor 120 may generate a plurality of intermediate images by stitching original images included in the same stitching group.

In one embodiment of the present disclosure, when the plurality of original images include the first to ninth original images as shown in FIG. 3 and the first stitching direction is a row direction, the image processor 120 performs the first stitching stitching the first to third original images included in the group to create a first intermediate image, stitching the fourth to sixth original images included in the second stitching group to create a second intermediate image, and third stitching A third intermediate image may be created by stitching the seventh to ninth original images included in the group.

In another embodiment of the present disclosure, when the plurality of original images include first to ninth original images as shown in FIG. 3 and the first stitching direction is a column direction, the image processor 120 performs the first stitching A first intermediate image is created by stitching the first, fourth, and seventh original images included in the group, and a second intermediate image is created by stitching the second, fifth, and eighth original images included in the second stitching group. A third intermediate image may be generated by stitching the third, sixth, and ninth original images included in the third stitching group.

In operation S440, the image processor 120 may create a merged image by stitching the plurality of intermediate images based on the second stitching direction.

In one embodiment of the present disclosure, when the plurality of original images include the first to ninth original images as shown in FIG. 3 and the second stitching direction is in the column direction, the image processor 120 performs row direction. A merged image may be generated by stitching first to third intermediate images that are stitched images.

In another embodiment of the present disclosure, when the plurality of original images include the first to ninth original images as shown in FIG. 3 and the second stitching direction is in the row direction, the image processor 120 performs the stitching in the column direction. A merged image may be generated by stitching first to third intermediate images that are stitched images.

5A to 5D are diagrams illustrating a process of generating a merged image from a plurality of original images by an imaging device according to an embodiment of the present disclosure.

5A to 5D, for convenience of illustration, there is no overlapping area between the plurality of original images, and the plurality of original images have the same angle and the same magnification, but in reality, there is an overlapping area between the plurality of original images. exists, and a plurality of original images may have different angles and different magnifications.

Referring to FIGS. 5A to 5D , a process of generating a merged image by stitching a plurality of original images by the image processor 120 may be confirmed. 5A to 5D illustrate an embodiment in which the first stitching direction is a column direction and the second stitching direction is a row direction. At this time, since the first stitching direction is the column direction, the image processor 120 classifies the first, fourth, and seventh original images I1, I4, and I7 into the first stitching group, and the second, fifth, and eighth The original images I2, I5, and I8 may be classified as a second stitching group, and the third, sixth, and ninth original images I3, I6, and I9 may be classified as a third stitching group.

First, referring to FIG. 5A , a process of stitching the first, fourth, and seventh original images I1, I4, and I7 included in the first stitching group by the image processor 120 can be confirmed.

As shown on the left side of FIG. 5A, in the state where the first to ninth original images I1 to I9 exist, the image processor 120 stitches the first original image I1 and the fourth original image I4 can do. In the state shown in the middle of FIG. 5A generated accordingly, the image processor 120 generates an image T1 in which the first original image I1 and the fourth original image I4 are stitched together and a seventh original image ( I7) may be stitched to generate the first intermediate image M1. Accordingly, a state as shown on the right side of FIG. 5A may be obtained.

Next, referring to FIG. 5B , a process of stitching the second, fifth, and eighth original images I2 , I5 , and I8 included in the second stitching group by the image processor 120 can be confirmed.

As shown on the left side of FIG. 5B, the first intermediate image M1 and the second, third, fifth, sixth, eighth, and ninth original images I2, I3, I5, I6, I8, and I9 are In the present state, the image processor 120 may stitch the second original image I2 and the fifth original image I5. In the state shown in the middle of FIG. 5B generated accordingly, the image processor 120 generates an image T2 in which the second original image I2 and the fifth original image I5 are stitched and an eighth original image ( I8) may be stitched to generate the second intermediate image M2. Accordingly, a state as shown on the right side of FIG. 5B may be obtained.

Next, referring to FIG. 5C , a process of stitching the third, sixth, and ninth original images I3, I6, and I9 included in the third stitching group by the image processor 120 can be confirmed.

As shown in the left side of FIG. 5C, in the state where the first and second intermediate images M1 and M2 and the third, sixth and ninth original images I3, I6 and I9 exist, the image processor 120 may stitch the third original image I3 and the sixth original image I6. In the state shown in the middle of FIG. 5C generated accordingly, the image processor 120 generates an image T3 where the third original image I3 and the sixth original image I6 are stitched together and a ninth original image ( I9) may be stitched to generate the third intermediate image M3. Accordingly, a state as shown on the right side of FIG. 5C may be obtained.

5A to 5C show an embodiment in which original images included in the same stitching group are stitched in the order of the first stitching group, the second stitching group, and the third stitching group, but the original images included in the same stitching group in a different order It is free even if it is stitched. For example, in another embodiment of the present disclosure, the image processor 120 may stitch original images included in the same stitching group in the order of the third stitching group, the first stitching group, and the second stitching group.

Finally, referring to FIG. 5D , an embodiment in which the image processor 120 stitches a plurality of intermediate images based on the second stitching direction can be confirmed.

As shown on the left side of FIG. 5D, in a state where the first to third intermediate images M1, M2, and M3 exist, the image processor 120 generates the first intermediate image M1 and the second intermediate image M2. can be stitched. In a state as shown in the middle of FIG. 5D generated accordingly, the image processor 120 generates an image T4 in which the first intermediate image M1 and the second intermediate image M2 are stitched together and the third intermediate image ( M3) may be stitched to generate the merged image F1. Accordingly, a merged image may be finally generated.

As such, since the imaging device 100 according to the present disclosure generates a merged image by stitching a plurality of original images based on the first and second stitching directions, images disposed in various directions may be stitched.

6 is a flowchart illustrating a method of stitching a plurality of original images by an imaging device according to an embodiment of the present disclosure.

Referring to FIG. 6 , in step S610, the image processor 120 may calculate a transformation matrix based on feature points between two images to be stitched.

The stitching target image is an image to be stitched by the image processor 120, and may be an original image, an image created by stitching the original image and the original image, an intermediate image, or an image created by stitching the intermediate image and the intermediate image.

The transformation matrix may be a matrix representing a mapping relationship or transformation relationship between two images to be stitched in order to offset a difference in rotation angle and magnification between the two images to be stitched.

The image processor 120 may calculate a transformation matrix based on feature points in an overlapping area between two images to be stitched. For example, if two images to be stitched are original images included in the same stitching group, the image processor 120 may calculate a transformation matrix based on feature points within an overlapping area between the original images included in the same stitching group. .

A more detailed method of calculating the transformation matrix by the image processor 120 will be described later with reference to FIG. 7 .

In step S620, the image processor 120 may detect a seam in an overlapping area between two images to be stitched.

The plant may be a boundary of a subject existing in the image. In this case, the image processor 120 may detect which part is the same part in the two stitching target images by detecting seams included in each of the two stitching target images.

A more detailed method of detecting the seam by the image processor 120 will be described later with reference to FIG. 8 .

In step S630, the image processor 120 may blend the two images to be stitched based on the transformation matrix and the seam.

The image processor 120 may warp the two stitching target images based on the transformation matrix, and blend the warped two stitching target images so that detected seams match.

7 is a flowchart illustrating a method of calculating a transformation matrix between a plurality of original images by an imaging device according to an embodiment of the present disclosure.

Referring to FIG. 7 , in step S710, the image processor 120 may determine whether a transformation matrix previously stored in the memory 130 exists.

When the image processor 120 generates a plurality of original images through the image sensor 110 while rotating the optical device 140 in a preset direction and angle, the rotation angle and magnification of the plurality of original images remain the same. It can be. In this case, transformation matrices between a plurality of original images can always be calculated identically. Accordingly, by storing the transformation matrix corresponding to the rotation information of the optical device 140 in the memory 130, the time required to generate the merged image may be reduced.

If there is a transformation matrix pre-stored in the memory 130 , in step S720 , the image processor 120 may use the transformation matrix pre-stored in the memory 130 instead of calculating the transformation matrix. That is, the image processor 120 may blend original images included in the same stitching group based on the transformation matrix read from the memory 130 .

When the transformation matrix pre-stored in the memory 130 does not exist, in step S730, the image processor 120 may cut out an overlapping region of the two images to be stitched. In this case, the image processor 120 may cut out an area obtained by adding a margin area to an overlapping area between the two images to be stitched.

In step S740, the image processor 120 may downscale the overlapping region cut out of the two images to be stitched by M times. In this case, M, which is a down-scaling multiple, may be set to such an extent that feature points can be identically detected in the two stitching target images.

In step S750, the image processor 120 may detect feature points in the overlapping region scaled down by M times between the two images to be stitched, and match the feature points of the two images to be stitched.

The feature point may be a point that has a large color difference from the surrounding area in the stitching target image. In an embodiment of the present disclosure, the image processor 120 may detect feature points in two images to be stitched and then match feature points having the same color in the two target images to be stitched.

In step S760, the image processor 120 may calculate a transformation matrix based on feature points of the two images to be stitched. In one embodiment of the present disclosure, the image processor 120 may calculate a transformation matrix such that areas surrounding feature points in two images to be stitched may be identically transformed.

As described above, since the imaging device 100 according to the present disclosure calculates the transformation matrix in the overlapping area, it is possible to generate a merged image with a small amount of calculation and a small processing time.

8 is a flowchart illustrating a method of detecting a seam between a plurality of original images by an imaging device according to an embodiment of the present disclosure.

Referring to FIG. 8 , in step S810, the image processor 120 may downscale two images to be stitched N times. In this case, N, which is a down-scaling multiple, may be set to such an extent that a mask can be calculated in two stitching target images.

In step S820, the image processor 120 may warp the two images to be stitched that are down-scaled by N times based on the transformation matrix. The image processor 120 selects one of the two stitching target images as a warping target image, multiplies the warping target image by a transformation matrix, and maintains the stitching target image that is not the warping target image as it is, thereby forming two stitching target images. You can warp them.

In step S830, the image processor 120 may calculate a mask from the two warped images to be stitched. In one embodiment of the present disclosure, the image processor 120 may calculate a mask so that a foreground and a background may be distinguished in two images to be stitched. In this case, the foreground of the two stitching target images may be displayed in black and the background in white.

In step S840, the image processor 120 may cut out an overlapping region of the two images to be stitched. In this case, the image processor 120 may cut out an area obtained by adding a margin area to an overlapping area between the two images to be stitched.

In step S850, the image processor 120 may detect seams in the cropped overlapping region. In an embodiment of the present disclosure, the image processor 120 may detect a boundary of a mask as a seam in an overlapping region of two images to be stitched.

As described above, since the imaging device 100 according to the present disclosure detects seams in the overlapping area, a merged image can be generated with a small amount of computation and a small processing time.

9 is a flowchart illustrating a method of blending a plurality of original images by an imaging device according to an embodiment of the present disclosure.

Referring to FIG. 9 , in step S910, the image processor 120 may warp two images to be stitched based on a transformation matrix. The image processor 120 selects one of the two stitching target images as a warping target image, multiplies the warping target image by a transformation matrix, and maintains the stitching target image that is not the warping target image as it is, thereby forming two stitching target images. You can warp them.

In step S920, the image processor 120 may perform correction on the two images to be stitched. The image processor 120 corrects the color, tone, light exposure, etc. centered on the region where the seams of the two warped stitching images are located, so that the images are more natural after the two stitching target images are stitched. can

In step S930, the image processor 120 may crop unnecessary regions from the two images to be stitched. In one embodiment of the present disclosure, the image processor 120 may crop an area in which no image is generated or an area not included in a merged image in two stitching target images.

In this case, if memory optimization is required, the image processor 120 may crop unnecessary regions from the two stitching target images and then blend the two stitching target images. In this way, by excluding unnecessary areas through cropping and blending, the amount of computation and memory usage can be reduced.

In step S940, the image processor 120 may blend the two images to be stitched. The image processor 120 may complete stitching of the two stitching target images by blending the two stitching target images around the detected feature points and seams.

As described above, since the imaging device 100 according to the present disclosure stitches a plurality of original images based on the transformation matrix calculated in the overlapping area and the seam detected in the overlapping area, a merged image can be generated with a small amount of calculation and a small processing time. there is.

10 is a block diagram illustrating an imaging device according to another exemplary embodiment of the present disclosure.

Referring to FIG. 10 , an imaging device 200 according to another embodiment of the present disclosure may include a memory 210, an image sensor 220, an image processor 230, an optical device 240, and a controller 250. can

At this time, the memory 210 of FIG. 10 is the memory 130 of FIG. 1, the image sensor 220 of FIG. 10 is the image sensor 110 of FIG. 1, and the image processor 230 of FIG. 10 is the image of FIG. The processor 120 and the optical device 240 of FIG. 10 may perform operations similar to those of the optical device 140 of FIG. 1 . Hereinafter, differences from the configurations of FIG. 1 will be mainly described.

The memory 210 may store a transformation matrix used for warping of the original image, a cropping region representing an unnecessary region in the original image, and an overlapping region representing an overlapping region of the original image with another original image.

When the imaging device 200 generates a merged image based on the plurality of original images, the transformation matrix is a mapping relationship or transformation between the plurality of original images in order to offset a difference in rotation angle and magnification between the plurality of original images. It can be a matrix representing a relationship. The transformation matrix may be pre-calculated based on feature points within an overlapping area between a plurality of original images and then stored in the memory 210 .

The cropping area may indicate an area that is not used for generating a merged image when the imaging device 200 generates a merged image based on a plurality of original images. The cropping area may be pre-calculated in consideration of a relationship between a plurality of original images and then stored in the memory 210 .

The overlapping area may indicate an area included in one original image that overlaps an area included in another original image. The overlapping area may be an area used to calculate feature points, mapping relationships, and transformation relationships when the imaging device 200 generates a merged image based on a plurality of original images. The overlapping area may be calculated in advance by considering the relationship between the plurality of original images and then stored in the memory 210 .

In this case, when the merged image is generated based on a plurality of original images arranged in both directions including the first direction and the second direction, the cropping area is the original image in any one of the first and second directions. It can be set to crop. In one embodiment of the present disclosure, the first direction may be a column direction, and the second direction may be a row direction. However, this is only an exemplary embodiment, and is not limited thereto, and the first direction may be a row direction and the second direction may be a column direction.

The image sensor 220 may generate an original image, generate a warped image by warping the original image using a transformation matrix, and generate a cropped image by cropping the warped image using a cropping region. In this specification, an embodiment in which the image sensor 220 generates a warped image and a cropped image is mainly described, but this is for convenience of description, and the technical spirit of the present disclosure is not limited thereto. That is, the image processor 230 may generate a warped image and a cropped image from the original image received from the image sensor 220, and in this embodiment, the memory 210 may generate the warped image and the cropped image. Information (for example, a transformation matrix, a cropping area, etc.) required for processing may be provided to the image processor 230 . Furthermore, the image sensor 220 and the image processor 230 may be integrated into a single circuit or chip.

In more detail, the image sensor 220 may generate an original image by photographing a target object. In this case, the image sensor 220 may generate an original image by photographing the target object upon receiving a photographing signal from the controller 250 .

The image sensor 220 may receive a transformation matrix from the memory 210 . At this time, the image sensor 220, prior to receiving the photographing signal from the controller 250, converts a transformation matrix corresponding to rotation information included in the rotation signal transmitted from the controller 250 to the optical device 240 from the memory 210. can receive The image sensor 220 may generate a warped image by warping the original image by multiplying the original image by the transformation matrix.

Then, the image sensor 220 may receive the cropping area from the memory 210 . At this time, the image sensor 220 stores the cropping area corresponding to the rotation information included in the rotation signal transmitted from the controller 250 to the optical device 240 prior to receiving the photographing signal from the controller 250 in the memory 210. can be received from The cropped area may be calculated to include an area not used for generating the merged image in the warped image and stored in the memory 210 . The image sensor 220 may generate a cropped image by cropping a cropped area in the warped image. In this case, the image sensor 220 may generate a cropped image by cropping pixels corresponding to the cropped area in the warped image.

Also, the image sensor 220 may receive an overlapping area from the memory 210 . Prior to receiving the shooting signal from the controller 250, the image sensor 220 receives an overlapping area corresponding to rotation information included in the rotation signal transmitted from the controller 250 to the optical device 240 from the memory 210. can do. The overlapping area may include coordinates of an overlapping area between a plurality of original images, and the image sensor 220 may use the overlapping area when calculating a relationship between the plurality of original images.

Also, the image processor 230 may calculate transformation matrices and detect seams included in the original image within the overlapping area. In this case, the image processor 230 may receive the overlapping area directly from the memory 210 or may receive the overlapping area from the image sensor 220 . For example, the image sensor 220 may load information about an overlapping area on a header or footer of an original image and transmit the information to the image processor 230 . As another example, the image sensor 220 may store the overlapping area received from the memory 210 in a storage area such as an internal register, and the image processor 230 may read the overlapping area stored in the image sensor 220. . However, the method of receiving the overlapping area by the image processor 230 is not limited thereto. In an embodiment of the present disclosure, when the image sensor 220 generates one original image, the cropped image is generated based on the original image. A cropping image may be transmitted to the image processor 230 by generating a . That is, whenever an original image is generated, the image sensor 220 may immediately generate a cropped image by performing warping and cropping and transmit the generated cropped image to the image processor 230 .

The image processor 230 may receive a plurality of cropped images from the image sensor 220 and generate a merged image by blending the plurality of cropped images. In this case, the image processor 230 may generate a merged image by blending a plurality of cropped images so that seams detected in the plurality of cropped images match.

In an embodiment of the present disclosure, the image processor 230 may generate a merged image based on N (N is a natural number equal to or greater than 2) cropped images received from the image sensor 220 . That is, when N preset cropped images are received from the image sensor 220, the image processor 230 may generate a merged image by blending the received N cropped images.

The optical device 240 may adjust the field of view of a plurality of original images by rotating according to rotation information including directions and angles. Rotation information of the optical device 240 may be set in advance, and the optical device 240 may equally rotate based on the same rotation information. In this case, as the optical device 240 rotates based on the same rotation information, rotation angles and magnifications between a plurality of original images generated by the image sensor 220 may be maintained the same.

In one embodiment of the present disclosure, the transformation matrix and cropping area stored in memory 210 may be calculated based on rotation information. The field of view of the original image generated by the image sensor 220 may be adjusted according to rotation of the optical device 240 based on rotation information. Since the original image is generated to have a different field of view according to the rotation information, the image sensor 220 must perform warping and cropping of the original image differently according to the rotation information. Thus, the transformation matrix and cropping area can be calculated based on the rotation information.

The controller 250 may transmit a rotation signal to the optical device 240 and transmit a photographing signal corresponding to the rotation signal to the image sensor 220 .

The rotation signal may be a signal for controlling rotation of the optical device 240 . In one embodiment of the present disclosure, the controller 250 may transmit a rotation signal including rotation information to the optical device 240, and the optical device 240 may respond to the rotation signal received from the controller 250. can rotate

The photographing signal may be a signal for controlling photographing of the image sensor 220 . At this time, the photographing signal may be transmitted to the image sensor 220 as the rotation of the optical device 240 according to the rotation signal is completed. In one embodiment of the present disclosure, the controller 250 may transmit a photographing signal to the image sensor 220, and the image sensor 220 photographs a target object in response to the photographing signal received from the controller 250. Original image can be created.

As described above, the imaging device 200 according to the present disclosure stores the transformation matrix calculated based on the rotation information of the optical device 240 in the memory 210, so that the image sensor 220 does not perform a complicated calculation and the image sensor 220 does not perform a complicated calculation. A warped image may be generated by warping an image using a transformation matrix, and a cropped image may be generated by cropping the warped image using a cropping region.

In addition, the imaging device 200 according to the present disclosure generates a merged image by performing warping and cropping through the image sensor 220 and blending through the image processor 230, thereby generating a merged image with less processing time. can create

11A to 11D are diagrams illustrating a process of generating a merged image from a plurality of original images by an imaging device according to another embodiment of the present disclosure.

Referring to FIG. 11A , a plurality of original images OI1 , OI2 , and OI3 generated by photographing the target object TO through the image sensor 220 may be checked.

The image sensor 220 may have different views by rotation of the optical device 240 and capture the target object TO. In the embodiment of FIG. 11A , the image sensor 220 has a first field of view S1 , a second field of view S2 , and a third field of view S3 by rotation of the optical device 240 and detects the target object TO. can be filmed The image sensor 220 captures the target object TO based on the first field of view S1 , the second field of view S2 , and the third field of view S3 , and obtains a first original image OI1 and a second original image OI1 . An image OI2 and a third original image OI3 may be acquired.

Referring to FIG. 11B , a plurality of warped images WI1 , WI2 , and WI3 generated by warping the plurality of original images OI1 , OI2 , and OI3 through the image sensor 220 may be identified.

The image sensor 220 receives a plurality of transformation matrices corresponding to each of the plurality of original images OI1, OI2, and OI3 from the memory 210, and converts the plurality of original images OI1, OI2, and OI3 into a plurality of A plurality of warped images WI1 , WI2 , and WI3 may be generated by performing warping using transformation matrices.

In the embodiment of FIG. 11B , the transformation matrix used for warping the first original image OI1 may rotate the first original image OI1 in a counterclockwise direction. Also, the transformation matrix used for warping the second original image OI2 may not rotate the second original image OI2. Finally, the transformation matrix used for warping the third original image OI3 may rotate the third original image OI3 clockwise.

In this case, the plurality of warped images WI1, WI2, and WI3 generated by warping the plurality of original images OI1, OI2, and OI3 may include an area that does not contain information as shown in gray in FIG. 11B. there is. This is an unnecessary area for generating a merged image and can be removed through cropping.

In this case, when the image sensor 220 generates one original image, it may generate a warped image based on the original image. For example, when the image sensor 220 generates the first original image OI1, the first warped image WI1 is obtained by warping the first original image OI1 regardless of the generation of the second original image OI2. ) can be created. Also, when the image sensor 220 generates the second original image OI2, regardless of the generation of the third original image OI3, the second original image OI2 is warped to obtain a second warped image WI2. can create

Referring to FIG. 11C , the plurality of cropped images CI1 , CI2 , and CI3 generated by cropping the plurality of warped images WI1 , WI2 , and WI3 through the image sensor 220 may be checked.

The image sensor 220 receives a plurality of cropping areas corresponding to each of the plurality of warped images WI1 , WI2 , and WI3 from the memory 210 , and generates a plurality of warped images WI1 , WI2 , and WI3 . A plurality of cropped images CI1 , CI2 , and CI3 may be generated by cropping using the cropping regions of .

In the embodiment of FIG. 11C , the cropping area may be an area outside of an area surrounded by a dotted-dashed line among areas within the plurality of warped images WI1 , WI2 , and WI3 . Accordingly, the image sensor 220 crops the plurality of warped images WI1 , WI2 , and WI3 along the dotted-dot chain line and generates a plurality of cropped images CI1 , CI2 , and CI3 in regions inside the dotted-dash line. can

In this case, when the image sensor 220 generates one warped image, it may generate a cropped image based on the warped image. For example, when the image sensor 220 generates the first warped image WI1, the first cropped image WI1 is cropped regardless of the generation of the second warped image WI2. (CI1) can be created. In addition, when the image sensor 220 generates the second warped image WI2, regardless of the generation of the third warped image WI3, the second warped image WI2 is cropped to obtain a second cropped image CI2. ) can be created.

Referring to FIG. 11D , a merged image MI1 generated by blending the plurality of cropped images CI1 , CI2 , and CI3 through the image processor 230 may be checked.

The image processor 230 receives a plurality of cropped images CI1 , CI2 , and CI3 from the image sensor 220 , and acquires a plurality of seams so that seams detected in the plurality of cropped images CI1 , CI2 , and CI3 match. The merged image MI1 may be generated by blending the cropped images CI1 , CI2 , and CI3 .

In the embodiment of FIG. 11D , the image processor 230 blends the plurality of cropped images CI1 , CI2 , and CI3 so that lines indicated by dashed-dotted lines in the plurality of cropped images CI1 , CI2 , and CI3 overlap each other. Thus, the merged image MI1 may be generated. For example, the image processor 230 overlaps the one-dotted chain line displayed in the first cropped image CI1 and the one-dotted chain line displayed in the second cropped image CI2 so that the uppermost line overlaps the first cropped image CI1. ) and the second cropping image CI2 may be blended. Further, the image processor 230 may overlap the second cropping image CI2 and the third dot-and-dotted line displayed on the third cropping image CI3 so that the line located at the bottom of the dashed-dotted line displayed on the second cropping image CI2 and the dashed-dotted line displayed on the third cropping image CI3 overlap. The cropping image CI3 may be blended.

In this case, when receiving N cropped images from the image sensor 220, the image processor 230 may generate a merged image MI1 by blending the received N cropped images. When N is 3 as in the embodiment of FIG. 11D , when receiving three cropped images from the image sensor 220, the image processor 230 may generate a merged image by blending the received three cropped images. there is.

12 is a flowchart illustrating a process of generating a merged image from a plurality of original images by an imaging device according to another embodiment of the present disclosure.

Referring to FIG. 12 , in step S1210, the imaging device 200 may generate an original image by photographing the target object through the image sensor 220. As the image sensor 220 receives a photographing signal from the controller 250, it may create an original image by photographing the target object.

In step S1220 , the imaging device 200 may generate a warped image through the image sensor 220 . The image sensor 220 may generate a warped image by warping the original image using the transformation matrix received from the memory 210 .

In operation S1230 , the imaging device 200 may generate a cropping image through the image sensor 220 . The image sensor 220 may generate a cropped image by cropping the warped image using the cropping area received from the memory 210 .

In step S1240, the imaging device 200 may generate a merged image through the image processor 230. The image processor 230 may generate a merged image by blending the N cropped images received from the image sensor 220 .

As described above, the imaging device 200 according to the present disclosure performs warping and cropping through the image sensor 220 and blending through the image processor 230 to generate a merged image, thereby obtaining a merged image with less processing time. can create

13 is a flowchart illustrating a process of generating an original image by an imaging device according to another embodiment of the present disclosure.

Referring to FIG. 13 , in step S1310 , the controller 250 may transmit a rotation signal to the optical device 240 . Upon receiving a rotation signal from the controller 250, the optical device 240 may rotate based on rotation information included in the rotation signal. Accordingly, the field of view of the image sensor 220 photographing the target object may be adjusted.

In step S1320, the controller 250 may transmit a photographing signal to the image sensor 220. When the rotation of the optical device 240 is completed, the controller 250 may transmit a photographing signal to the image sensor 220 . In one embodiment of the present disclosure, the controller 250 may transmit a photographing signal to the image sensor 220 upon receiving a rotation completion signal from the optical device 240 . In another embodiment of the present disclosure, the controller 250 transmits a rotation signal to the optical device 240, and then, when the maximum time required for rotation of the optical device 240 elapses, the image sensor 220 takes pictures. signal can be transmitted.

In step S1330, the image sensor 220 may create an original image by photographing the target object. The image sensor 220 may photograph a target object in response to a photographing signal received from the controller 250 and generate an original image. The image sensor 220 may start capturing an image in synchronization with an input capturing signal, capture one frame or a plurality of frames, and generate an original image. Also, the image sensor 220 may capture another frame or a plurality of frames in synchronization with an additionally input capturing signal. That is, the image sensor 220 may generate one or a plurality of original images synchronized with each photographing signal input from the outside. At this time, step S1330 may correspond to step S1210 of FIG. 12 .

14 is a flowchart illustrating a process of generating a merged image from a plurality of original images arranged in both directions by an imaging device according to another embodiment of the present disclosure.

Referring to FIG. 14 , in step S1410 , the imaging device 200 may capture an original image through the image sensor 220 . The image sensor 220 may capture an original image as it receives a capture signal from the controller 250 . Step S1410 may be the same as step S1210 of FIG. 12 .

In step S1420 , the imaging device 200 may generate a warping image through the image sensor 220 . The image sensor 220 may generate a warped image by warping the original image using the transformation matrix received from the memory 210. Step S1420 may be the same as step S1220 of FIG. 12 .

In step S1430 , the imaging device 200 may generate a cropped image through the image sensor 220 . The image sensor 220 may generate a cropped image by cropping the warped image using the cropping area received from the memory 210 .

When generating a merged image from a plurality of original images arranged in both directions, the cropping area may be set to crop the warped image in one direction selected from among the first and second directions. For example, when original images are arranged in both directions as in the embodiments shown in FIGS. 5A to 5D , the image processor 230 generates a plurality of intermediate images through stitching based on a first stitching direction, and second A merged image may be generated through stitching based on a stitching direction. In this case, the plurality of intermediate images may have different angles and different magnifications. Accordingly, when performing stitching based on the second stitching direction, the image processor 230 must warp and crop the plurality of intermediate images to match rotation angles and magnifications. In this situation, if the image sensor 220 crops the original images based on the cropping area set to be cropped in both directions, the images are duplicated in the same direction and included in the merged image. area may be reduced. Therefore, when generating a merged image from a plurality of original images arranged in both directions, by setting the cropping area so that the warped image is cropped in one direction selected from among the first and second directions, a transitional image compared to the original image is generated. It is possible to prevent generation of a merged image in which a large number of areas are deleted.

In step S1440, the imaging device 200 may generate a plurality of intermediate images through the image processor 230. The image processor 230 may generate an intermediate image by blending the N cropped images received from the image sensor 220 . In this case, the N cropping images may be arranged in the same direction. For example, if the N cropped images are cropped images generated based on the cropping area set to be cropped in the first direction in step S1430, the N cropped images are N original images arranged in the first direction. It may be an image generated based on.

In step S1450, the imaging device 200 may generate a plurality of intermediate warped images through the image processor 230. The image processor 230 may generate a plurality of intermediate warped images by warping the plurality of intermediate images using the transformation matrix received from the memory 210 .

In this case, warping of the original image in step S1420 is performed by the image sensor 220 , but warping of the intermediate image in step S1450 may be performed by the image processor 230 . In this way, by performing operations after generating the intermediate image by the image processor 230, the time required for image processing may be reduced.

In step S1460, the imaging device 200 may generate a plurality of intermediate cropped images through the image processor 230. The image processor 230 may generate a plurality of intermediate cropped images by cropping the plurality of intermediate warped images using the cropping region received from the memory 210 .

At this time, the cropping of the warped image in step S1430 is performed by the image sensor 220, but the cropping of the intermediate warped image in step S1460 may be performed by the image processor 230. In this way, by performing operations after generating the intermediate image by the image processor 230, the time required for image processing may be reduced.

In step S1470, the imaging device 200 may generate a merged image through the image processor 230. The image processor 230 may generate a merged image by blending a plurality of intermediate cropped images.

15 is a diagram illustrating a process of generating a cropped image by cropping in one direction by an imaging device according to another embodiment of the present disclosure.

Referring to FIG. 15 , a plurality of cropped images CI1 , CI2 , CI3 generated by the imaging device 200 by cropping the plurality of warped images WI1 , WI2 , WI3 in only one direction and an intermediate image (M1) can be confirmed. This may be an operation corresponding to steps S1430 and S1440 of FIG. 14 .

As described above with reference to FIG. 14 , when the merged image is generated based on a plurality of original images arranged in both directions, the cropping area moves the warped image in one direction selected from among the first and second directions. It can be set to crop.

In the embodiment of FIG. 15 , the cropped area is set so that the warped image is cropped only in the column direction, and may be an area outside of the area surrounded by the dotted-dashed line among the areas within the plurality of warped images WI1, WI2, and WI3. . Accordingly, the image sensor 220 crops the plurality of warped images WI1 , WI2 , and WI3 along the dotted-dot chain line and generates a plurality of cropped images CI1 , CI2 , and CI3 in regions inside the dotted-dash line. can

Next, the image processor 230 receives the plurality of cropped images CI1 , CI2 , and CI3 from the image sensor 220 , and the seams detected in the plurality of cropped images CI1 , CI2 , and CI3 match. The intermediate image M1 may be generated by blending the plurality of cropped images CI1 , CI2 , and CI3 to

In the embodiment of FIG. 15 , the image processor 230 blends the plurality of cropped images CI1 , CI2 , and CI3 so that lines indicated by dashed-dotted lines in the plurality of cropped images CI1 , CI2 , and CI3 overlap each other. Thus, an intermediate image M1 may be generated.

When generating a merged image from a plurality of original images arranged in both directions, by setting the cropping area so that the warped image is cropped in one direction selected from among the first and second directions, compared to the original image It is possible to prevent generation of a merged image in which excessively large areas are deleted.

16 is a block diagram illustrating an imaging device according to another exemplary embodiment of the present disclosure.

Referring to FIG. 16 , an imaging device 300 according to another embodiment of the present disclosure includes a memory 310, an image sensor 320, an application processor (AP) 330, an optical device 340, and a controller 350. ) may be included.

At this time, the memory 310, the image sensor 320, the optical device 340, and the controller 350 may be the same as the memory 210, the image sensor 220, the optical device 240, and the controller 250 of FIG. 10. can

The AP 330 may be a central processing unit (CPU), a microprocessor, or a microcontroller unit (MCU), but is not limited thereto.

AP 330 may include an image processor. In this case, the AP 330 of the imaging device 300 may be a processor that executes software for performing image processing, and the image processor may be implemented as software or a combination of hardware and software.

17A and 17B are block diagrams illustrating an imaging device including a plurality of imaging devices according to an exemplary embodiment of the present disclosure.

Referring to FIG. 17A , an electronic device 20000 may include a multi-imaging device 1100, an AP 4000, and a memory 5000. The memory 5000 may perform the same function as the memory 130 shown in FIG. 1, and thus duplicate descriptions are omitted. At least one of the imaging devices 1100a, 1100b, and 1100c of FIG. 17A may perform a function similar to that of the imaging device 100 of FIG. 1 .

The electronic device 20000 may capture and/or store an image of a subject using a CMOS image sensor, and may be implemented as a mobile phone, a tablet computer, or a portable electronic device. Portable electronic devices may include laptop computers, mobile phones, smart phones, tablet PCs, wearable devices, and the like. The electronic device 20000 may include one or more imaging devices and an AP processing image data generated by the one or more imaging devices.

The multi-imaging device 1100 may include a first imaging device 1100a, a second imaging device 1100b, and a third imaging device 1100c. For convenience of explanation, three imaging devices 1100a to 1100c are shown, but the multi-imaging device 1100 may include a variety of imaging devices without being limited thereto.

Imaging devices included in the multi-imaging device 1100 may generate a merged image by stitching a plurality of original images based on first and second stitching directions. Imaging devices included in the multi-imaging device 1100 may stitch a plurality of original images based on the transformation matrix calculated in the overlapping area and the seams detected in the overlapping area. Accordingly, images disposed in various directions may be stitched and a merged image may be generated with a small amount of calculation and a small processing time.

In addition, imaging devices included in the multi-imaging device 1100 may store a transformation matrix calculated based on rotation information of an optical device in a memory. Also, the imaging devices included in the multi-imaging device 1100 may generate a merged image by performing warping and cropping through an image sensor and blending through an image processor. Accordingly, the image sensor can generate a warped image by warping the original image using a transformation matrix without performing complicated calculations, and cropping the warped image using a cropping region to generate a cropped image. You can create merged images with processing time.

Hereinafter, a detailed configuration of the imaging device 1100b will be described in more detail with reference to FIG. 17B , but the following description may be equally applied to other imaging devices 1100a and 1100c according to embodiments.

Referring to FIG. 17B , the second imaging device 1100b includes a prism 1105, an optical path folding element (OPFE) 1110, an actuator 1130, an image sensing device 1140, and a storage unit. (1150).

The prism 1105 may include a reflective surface 1107 of a light reflective material to change a path of light L incident from the outside.

According to an exemplary embodiment, the prism 1105 may change a path of light L incident in the first direction X to a second direction Y perpendicular to the first direction X. In addition, the prism 1105 rotates the reflective surface 1107 of the light reflecting material in the direction A around the central axis 1106 or rotates the central axis 1106 in the direction B to move in the first direction X. A path of the incident light L may be changed in a second direction Y, which is perpendicular to the second direction Y. At this time, the OPFE 1110 may also move in a third direction (Z) perpendicular to the first direction (X) and the second direction (Y).

In an exemplary embodiment, as shown, the maximum angle of rotation of the prism 1105 in the A direction is less than or equal to 21 degrees in the positive A direction and greater than 21 degrees in the negative A direction. However, the embodiments are not limited thereto.

In an exemplary embodiment, the prism 1105 can move about 20 degrees in the plus or minus B direction, or between 10 and 20 degrees, or between 15 and 20 degrees, where the angle of movement is It may move at the same angle in the plus (+) or minus (-) B direction, or may move to an almost similar angle within a range of 1 degree, but is not limited thereto.

In an exemplary embodiment, the prism 1105 may move the reflective surface 1106 of the light reflecting material in a third direction (eg, Z direction) parallel to the extension direction of the central axis 1106 .

The OPFE 1110 may include, for example, optical lenses consisting of m (where m is a natural number) groups. The m lenses may move in the first direction X to change the optical zoom ratio of the imaging device 1100b. For example, assuming that the basic optical zoom magnification of the imaging device 1100b is Z, when m optical lenses included in the OPFE 1110 are moved, the optical zoom magnification of the imaging device 1100b is 3Z or 5Z or higher. It can be changed with optical zoom magnification.

The actuator 1130 may move the OPFE 1110 or an optical lens (hereinafter referred to as an optical lens) to a specific position. For example, the actuator 1130 may adjust the position of the optical lens so that the image sensor 1142 is positioned at the focal length of the optical lens for accurate sensing.

The image sensing device 1140 may include an image sensor 1142 , a control logic 1144 , and a memory 1146 . The image sensor 1142 may sense an image of a sensing target using light L provided through an optical lens. Since the image sensor 1142 of FIG. 17B may be functionally similar to the image sensor 110 of FIG. 1 or the image sensor 220 of FIG. 10 , duplicate descriptions will be omitted. The control logic 1144 may control overall operations of the second imaging device 1100b. For example, the control logic 1144 may control the operation of the second imaging device 1100b according to a control signal provided through the control signal line CSLb.

The memory 1146 may store information necessary for the operation of the second imaging device 1100b, such as calibration data 1147. The calibration data 1147 may include information necessary for the second imaging device 1100b to generate image data using light L provided from the outside. The calibration data 1147 is, for example, information about the degree of rotation described above, information about the focal length, information about the optical axis, and calibration information necessary for image processing etc. may be included. If the second imaging device 1100b is implemented in the form of a multi-state camera whose focal length changes according to the position of the optical lens, the calibration data 1147 is the focal length of each position (or state) of the optical lens. values and information related to auto focusing.

The storage unit 1150 may store image data sensed through the image sensor 1142 . The storage unit 1150 may be disposed outside the image sensing device 1140 and may be implemented in a stacked form with a sensor chip constituting the image sensing device 1140 . In an exemplary embodiment, the storage unit 1150 may be implemented as an electrically erasable programmable read-only memory (EEPROM), but the embodiments are not limited thereto.

In an exemplary embodiment, each of the plurality of imaging devices 1100a, 1100b, and 1100c may include an actuator 1130. Accordingly, each of the plurality of imaging devices 1100a, 1100b, and 1100c may include the same or different calibration data 1147 according to the operation of the actuator 1130 included therein.

In an exemplary embodiment, one imaging device (eg, the second imaging device 1100b) among the plurality of imaging devices 1100a, 1100b, and 1100c includes the prism 1105 and the OPFE 1110 described above. It is a folded lens type imaging device, and the remaining imaging devices (eg, 1100a and 1100b) are vertical type imaging devices that do not include the prism 1105 and the OPFE 1110. However, the embodiments are not limited thereto.

In an exemplary embodiment, one imaging device (eg, the third imaging device 1100c) among the plurality of imaging devices 1100a, 1100b, and 1100c uses IR (Infrared Ray) to detect depth. It may be a vertical type depth camera that extracts (depth) information. In this case, the AP 4000 merges image data provided from the depth camera with image data provided from another imaging device (eg, the first imaging device 1100a or the second imaging device 1100b). ) to generate a 3D depth image.

In an exemplary embodiment, at least two imaging devices (eg, the first imaging device 1100a or the second imaging device 1100b) among the plurality of imaging devices 1100a, 1100b, and 1100c have different fields of view. (Field of View, angle of view). In this case, for example, optical lenses of at least two imaging devices (eg, the first imaging device 1100a or the second imaging device 1100b) among the plurality of imaging devices 1100a, 1100b, and 1100c They may be different from each other, but are not limited thereto. For example, among the plurality of imaging devices 1100a, 1100b, and 1100c, the first imaging device 1100a may have a smaller field of view than the second and third imaging devices 1100b and 1100c. However, the multi-imaging device 1100 is not limited thereto, and the multi-imaging device 1100 may further include an imaging device having a larger field of view (FOV) than the originally used imaging devices 1100a, 1100b, and 1100c.

Also, in some embodiments, each of the plurality of imaging devices 1100a, 1100b, and 1100c may have different viewing angles. In this case, optical lenses included in each of the plurality of imaging devices 1100a, 1100b, and 1100c may also be different from each other, but are not limited thereto.

In some embodiments, each of the plurality of imaging devices 1100a, 1100b, and 1100c may be disposed physically separated from each other. That is, the sensing area of one image sensor 1142 is not divided and used by the plurality of imaging devices 1100a, 1100b, and 1100c, but the sensing area of each of the plurality of imaging devices 1100a, 1100b, and 1100c is independently An image sensor 1142 may be disposed.

The AP 4000 may include a plurality of subprocessors 4100a, 4100b, and 4100c, a decoder 4200, an imaging device controller 4300, a memory controller 4400, and an internal memory 4500.

The AP 4000 may be implemented separately from the plurality of imaging devices 1100a, 1100b, and 1100c. For example, the AP 4000 and the plurality of imaging devices 1100a, 1100b, and 1100c may be separately implemented as separate semiconductor chips.

Image data generated from each of the imaging devices 1100a, 1100b, and 1100c may be provided to corresponding sub-processors 4100a, 4100b, and 4100c through image signal lines ISLa, ISLb, and ISLc separated from each other. For example, image data generated by the first imaging device 1100a may be provided to the first sub-processor 4100a through the first image signal line ISLa, and image data generated by the second imaging device 1100b may be provided. Image data is provided to the second sub-processor 4100b through the second image signal line ISLb, and image data generated from the third imaging device 1100c is provided to the third sub-processor 4100b through the third image signal line ISLc. It may be provided to the processor 4100c. Such image data transmission may be performed using, for example, a Camera Serial Interface (CSI) based on MIPI (Mobile Industry Processor Interface), but embodiments are not limited thereto.

In an exemplary embodiment, one subprocessor may be arranged to correspond to a plurality of imaging devices. For example, the first sub-processor 4100a and the third sub-processor 4100c are not implemented separately from each other as shown, but integrated into one sub-processor, and the imaging device 1100a and the imaging device 1100c are implemented. Image data provided from ) may be selected through a selection element (eg, a multiplexer) and the like, and then provided to the integrated sub image processor.

The imaging device controller 4300 may provide a control signal to each of the imaging devices 1100a, 1100b, and 1100c. Control signals generated from the imaging device controller 1216 may be provided to the corresponding imaging devices 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

One of the plurality of imaging devices 1100a, 1100b, and 1100c is designated as a master camera (eg, 1100b) according to image generation information including a zoom signal or a mode signal, and the remaining imaging devices ( For example, 1100a and 1100c) may be designated as slave cameras. Such information may be included in a control signal and provided to the corresponding imaging devices 1100a, 1100b, and 1100c through separate control signal lines CSLa, CSLb, and CSLc.

Under the control of the imaging device controller 4300, the imaging devices 1100a, 1100b, and 1100c operating as masters and slaves may be changed. For example, when the viewing angle of the first imaging device 1100a is wider than that of the second imaging device 1100b and the zoom factor indicates a low zoom magnification, the second imaging device 1100b may operate as a master. , the first imaging device 1100a may operate as a slave. Conversely, when the zoom factor indicates a high zoom magnification, the first imaging device 1100a may operate as a master and the second imaging device 1100b may operate as a slave.

In an exemplary embodiment, a control signal provided from the imaging device controller 4300 to each of the imaging devices 1100a, 1100b, and 1100c may include a sync enable signal. For example, when the second imaging device 1100b is a master camera, and the first and third imaging devices 1100a and 1100c are slave cameras, the imaging device controller 4300 may be configured for the second imaging device 1100b. A sync enable signal may be transmitted. Upon receiving such a sync enable signal, the second imaging device 1100b generates a sync signal based on the provided sync enable signal, and transmits the generated sync signal to the first sync signal line SSL. and the third imaging devices 1100a and 1100c. The first imaging device 1100b and the second and third imaging devices 1100a and 1100c may transmit image data to the AP 4000 in synchronization with the sync signal.

As above, exemplary embodiments have been disclosed in the drawings and specifications. Embodiments have been described using specific terms in this specification, but they are only used for the purpose of explaining the technical spirit of the present disclosure, and are not used to limit the scope of the present disclosure described in the meaning or claims. Therefore, those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical scope of protection of the present disclosure should be determined by the technical spirit of the appended claims.

Claims (10)

an image sensor generating a plurality of original images; and
An image processor performing an operation to generate a merged image based on the plurality of original images;
The image processor
Classifying the plurality of original images into a plurality of stitching groups based on a first stitching direction, generating a plurality of intermediate images by stitching original images included in the same stitching group, and based on a second stitching direction characterized in that the merged image is generated by stitching the plurality of intermediate images
imaging device. According to claim 1,
Characterized in that the first and second stitching directions are a row direction or a column direction
imaging device. According to claim 2,
The imaging device,
Further comprising an optical device for adjusting the field of view of the plurality of original images by rotating according to rotation information including a direction and an angle,
The image processor,
Characterized in that determining the first and second stitching directions based on the rotation information
imaging device. According to claim 1,
The image processor,
The same stitching by blending original images included in the same stitching group based on a transformation matrix calculated in an overlapping area between original images included in the same stitching group and a seam detected in the overlapping area. Characterized in that the original images included in the group are stitched
imaging device. According to claim 4,
The image processor,
Characterized in that the transformation matrix is calculated based on feature points in an overlapping area between original images included in the same stitching group
imaging device. According to claim 1,
The image processor,
Characterized in that the plurality of intermediate images are stitched by blending the plurality of intermediate images based on a transformation matrix calculated in an overlapping region between the plurality of intermediate images and a seam detected in the overlapping region.
imaging device. An imaging device for generating a merged image based on a plurality of original images,
a memory that stores a transformation matrix used for warping the original image and a cropping region representing an unnecessary region in the original image;
an image sensor configured to generate the original image, generate a warped image by warping the original image using the transformation matrix, and generate a cropped image by cropping the warped image using the cropping region; and
An image processor receiving a plurality of cropped images from the image sensor and generating a merged image by blending the plurality of cropped images
imaging device. According to claim 7,
Characterized in that the cropping area is calculated to include an area not used for generating the merged image in the warping image.
imaging device. According to claim 7,
When the image sensor generates one original image, the cropped image is generated based on the original image and the cropped image is transmitted to the image processor;
Characterized in that the image processor generates the merged image based on N (N is a natural number of 2 or more) cropping images received from the image sensor
imaging device. According to claim 7,
When the merged image is generated based on a plurality of original images arranged in both directions including a first direction and a second direction, the cropping area converts the warped image to any selected one of the first and second directions. Characterized in that it is set to be cropped in one direction
imaging device.

Images