JP2017038292A - Image processing system, image processing method, image processing program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system which sequentially sticks images moving in X and Y-axis directions and a Z-axis direction to display a panoramic image on a real time basis.SOLUTION: The problem is solved by image processing comprising: moving means of a subject or a camera; image acquisition means; moving direction determination means which specifies coordinates where frequencies and luminances are matched for a first image and a second image, sets the specified coordinates to be applied to the same part of the subject, determines that the subject relatively moves to X and Y-axis directions when the coordinates are different, and determines that the subject relatively moves to the Z-axis direction when the coordinates are matched; means for matching the specified coordinates in the X and Y-axis directions and sequentially sticking them when the subject moves in the X and Y-axis directions; and means for calculating a difference of the luminances between pixels of the first image and the second image and sequentially sticking the pixel with the coordinate of the large difference to the first image when the subject moves in the Z-axis direction.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing for performing image processing to display images as panoramic images on a display in real time by sequentially pasting images captured by the camera while moving the subject and the camera relative to each other. It relates to a program and a recording medium.

Patent Document 1 discloses a system for creating an omnifocal image of an object, an imaging device that sequentially images the object in a hierarchical manner, a mounting table on which the object is mounted, and an interval between the imaging device and the mounting table. An interval adjustment unit that can adjust the image, an image converter that converts an image captured by the imaging device into data, an image processing unit that processes captured data output from the image converter, and an output from the image processing unit Display means for displaying the obtained image data, and the image brightness value obtained by comparing the image brightness value of each pixel obtained last time and the image brightness value of each pixel photographed this time. A technique is disclosed in which the latest omnifocal image data obtained by the image processing unit reconstructed based on the image processing unit is sequentially updated and displayed on the display unit as an image.

In addition, as an image processing unit, it is disclosed that an operation for emphasizing an outline of an illustration by an illustration creation support method disclosed in Japanese Patent No. 2775121 is performed.

Patent Document 2 discloses an image processing apparatus that performs image processing on image data of a plurality of original images using each of a plurality of images obtained by continuous shooting of a moving subject as an original image, and the subject is photographed. Based on the sequence and the position information of the image data of the subject in the image data of the original image. The shape of the group of pixels generated based on the correlation based on at least the color of the pixels included in the image data of the image, the pixel data of the future subject in terms of time, and the past subject in terms of time An image processing device that changes the stroke of the pixel data so as to be rough is disclosed.

JP 2003-281501 A JP 2011-176557 A

The invention of Patent Document 1 is a technique in which shooting of an object is performed by moving only the Z-axis direction without changing the XY-axis direction, and the captured images are pasted together. Therefore, when the camera is moved in the XY-axis direction, the image is captured. There was a problem that panorama images could not be seen on the display in real time.

In addition, since the process of comparing and calculating the image luminance value is a process of setting a threshold value and extracting a contour line, it is not a technique for sharpening the entire image, so there is a problem that an unclear portion is generated in the image. .

The invention of Patent Document 2 is a technique for creating a composite image based on positional information of image data before and after a subject moving in the XY axis direction. When the camera is moved in the Z axis direction, image pasting is performed. There is a problem that the panoramic image of the three-dimensional subject cannot be viewed on the display in real time.

Also, image processing that changes the touch of the paint brush that forms the color tone, lightness, and darkness of the pixel data of the future subject in time, and changes the touch to make the touch of the pixel data of the past subject temporally rough. Therefore, there is a problem that images with different sharpness are displayed on the image on the display.

Accordingly, an object of the present invention is to display a clear panoramic image in real time on a display by sequentially pasting together images captured by a camera while moving an object in the XY axis direction or the Z axis direction with the camera. An image processing apparatus, an image processing method, an image processing program, and a recording medium are provided.

  In the present invention, a panoramic image is a continuous image in which images of a subject taken by the camera while the subject and the camera are moved relative to each other in the XY axis direction or the Z axis direction are sequentially pasted in the horizontal direction or the vertical direction. Means.

The image processing apparatus 1 according to claim 1 performs image processing for sequentially displaying images captured by the camera while moving the subject and the camera relative to each other and displaying the panorama image on the display in real time. An image processing apparatus 1 that performs a moving unit 4 that moves a mounted subject or camera in an XY-axis direction or a Z-axis direction, captures the subject while moving the moving unit 4, and continuously acquires images. The image acquisition means 3, the frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state where a plurality of acquired image data are pasted together, and 1 of the first image data Compare the frequency component of the luminance component for each coordinate of the second image data acquired after the frame with the frequency analysis frequency, and select the coordinate range where the frequency difference for each coordinate is minimized. And searching for a coordinate that minimizes the difference in luminance of the corresponding coordinate range in the search of the respective image data, and using the searched coordinates of the subject in the first image data and the second image data. The coordinates of the corresponding parts are respectively specified, and when the coordinates of the specified first image data and the coordinates of the second image data are different, it is determined that the subject has relatively moved in the XY axis direction, and the coordinates match. The moving direction determining means for determining that the subject has moved relative to the Z-axis direction, and the moving direction determining means to determine that the subject has moved relative to the XY-axis direction. XY axis direction pasting means for matching the image data and the coordinates specified by the moving direction judging means in the XY axis direction and sequentially pasting them in time series;
When the moving direction determining means determines that the subject has moved relatively in the Z-axis direction, the luminance difference between the pixels for each coordinate range consisting of several pixel ranges of the first image data and the second image data. Z is calculated, and the respective differences are compared for each pixel having the same coordinate, and the pixels having the larger difference in the second image data are sequentially pasted to the first image data in time series. And an axial direction laminating means.

An image processing apparatus according to a second aspect is the image processing apparatus according to the first aspect, wherein the moving direction determination unit searches the coordinate range in which the difference in the frequency of the luminance component is minimized, The image was reduced to the first and second reduced image data, and the frequency analysis was performed on the luminance component for each coordinate of the first reduced image data, and the frequency component was analyzed for the luminance component for each coordinate of the second reduced image data. The frequency range is compared, and a coordinate range in which the difference between the frequencies is minimized is searched for.

  The image processing method 10 according to claim 3 is an image for performing image processing for displaying images as panoramic images on a display in real time by sequentially pasting together images captured by the camera while moving the subject and the camera relative to each other. A processing method for moving a placed subject or camera in an XY-axis direction or a Z-axis direction; and an image acquisition step for continuously capturing images by capturing the subject while moving by the moving means; The frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state in which a plurality of image data is pasted together, and the second acquired after one frame of the first image data The luminance component for each coordinate of the image data is compared with the frequency obtained by frequency analysis, and a coordinate range where the frequency difference for each coordinate is minimized is searched. In the search of the respective image data, a search is made for a coordinate that minimizes the difference in luminance of the corresponding coordinate range, and the searched coordinates are used for the matching part of the subject in the first image data and the second image data. When the coordinates of the identified first image data and the coordinates of the second image data are different from each other, it is determined that the subject has moved relatively in the XY axis direction, and when the coordinates match, the subject is If the moving direction determining step determines that the subject has moved relatively in the Z axis direction, and the moving direction determining means determines that the subject has moved relatively in the XY axis direction, the first image data and the second image data are obtained. An XY-axis direction pasting step of matching the coordinates specified by the movement direction determination unit in the XY-axis direction and sequentially bonding them in time series, and the subject in the Z-axis direction by the movement direction determination unit. If it is determined that the pixel has moved, the difference in luminance between each pixel is calculated for each coordinate range consisting of several pixel ranges of the first image data and the second image data, and each difference is used as a pixel having the same coordinate. And a Z-axis direction pasting step of sequentially pasting pixels of coordinates having a large difference in the second image data to the first image data in time series. To do.

  According to a fourth aspect of the present invention, there is provided the image processing method according to the third aspect, wherein in the moving direction determining step, the search of the coordinate range in which the difference in the frequency of the luminance component is minimized is performed for the first and second image data. The image was reduced to the first and second reduced image data, and the frequency analysis was performed on the luminance component for each coordinate of the first reduced image data, and the frequency component was analyzed for the luminance component for each coordinate of the second reduced image data. The frequency range is compared, and a coordinate range in which the difference between the frequencies is minimized is searched for.

  6. The image processing program according to claim 5, wherein the image processing program performs image processing for sequentially displaying images captured by the camera in time series while relatively moving the subject and the camera and displaying the images as a panoramic image on the display in real time. A moving procedure for moving a placed subject or camera in the XY-axis direction or the Z-axis direction, an image acquisition procedure for capturing an image of the subject while moving it by the moving means, and acquiring the image continuously; The frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state in which the plurality of image data are pasted together, and the second acquired after one frame of the first image data Compare the luminance component for each coordinate of the image data with the frequency analyzed frequency, and search for the coordinate range where the frequency difference for each coordinate is minimized Further, in the search of the respective image data, a search is made for a coordinate that minimizes the difference in luminance of the corresponding coordinate range, and the searched coordinates match the subject in the first image data and the second image data. If the coordinates of the first image data and the coordinates of the second image data are different from each other, it is determined that the subject has moved relatively in the XY axis direction, and the coordinates match. If the moving direction determination procedure determines that the subject has moved relative to the Z axis, and the moving direction determination means determines that the subject has moved relative to the XY axis, the first image data and the second image data XY-axis direction combining procedure in which the coordinates specified by the moving direction determining means are matched in the XY-axis direction and sequentially bonded in time series, and the subject is moved to the Z-axis by the moving direction determining means. When it is determined that the relative movement is made in the direction, the luminance difference between each pixel is calculated for each coordinate range consisting of several pixel ranges of the first image data and the second image data, and the differences are the same. A Z-axis direction combining procedure for comparing each pixel of coordinates and sequentially combining pixels of the second image data having a large difference to the first image data in time series. It is characterized by.

  According to a sixth aspect of the present invention, there is provided the image processing program according to the fifth aspect, wherein the search for the coordinate range in which the difference in the frequency of the luminance component is minimized in the moving direction determination procedure is performed for the first and second image data. The image was reduced to the first and second reduced image data, and the frequency analysis was performed on the luminance component for each coordinate of the first reduced image data, and the frequency component was analyzed for the luminance component for each coordinate of the second reduced image data. The frequency range is compared, and a coordinate range in which the difference between the frequencies is minimized is searched for.

  The image processing program according to claim 7 is the image processing program according to claim 5 or 6, wherein the entire image of the images sequentially combined in time series by the XY-axis direction combining procedure or the Z-axis direction combining procedure has an aspect ratio. An image display procedure for displaying the image so as to fit the screen size of the display in a fixed state is provided.

  A recording medium according to an eighth aspect is characterized in that the image processing program according to any one of the fifth to seventh aspects is recorded so as to be readable by a computer.

The invention according to any one of claims 1 to 8, wherein the image captured by the camera is sequentially pasted in time series in real time while the subject and the camera are shifted in the XY axis direction and the Z axis direction in correlation. A clear image of the object can be displayed on the display as a panoramic image including the depth in real time.

Since it is possible to instantly determine whether the relative positional relationship between the subject and the camera has moved in the XY-axis direction or the Z-axis direction, a clear image can be obtained in real time even for a three-dimensional subject. Can be displayed on the display.

According to the second, fourth, and sixth aspects of the present invention, the frequency analysis of the luminance can be performed in a very short time by performing the luminance frequency analysis in a state where the image is reduced. This further shortened the image processing time.

According to the seventh aspect of the present invention, since all the images of the subject within the range captured by the camera can be displayed on the display, the entire range captured by the camera of the subject can be always grasped.

It is a schematic diagram of the image processing apparatus according to claim 1 of the present invention. FIG. 4 is an explanatory diagram of pasting of a first image and a second image, (a) is a diagram of a subject, and (b) is a diagram showing a shooting range for the subject of the first image and the second image. It is explanatory drawing of the bonding of a 1st image and a 2nd image of the XY-axis direction, (a) is explanatory drawing of a 1st image, (b) is explanatory drawing of a 2nd image, (c) FIG. 7 is an explanatory diagram showing a cut-out portion of the second image that overlaps the first image, and (d) is a diagram in which the first image and the second image are bonded together. It is explanatory drawing of the search of the minimum coordinate range of the difference of each frequency of a 1st image and a 2nd image, (a) is description explaining the coordinate range in which the frequency of a 1st image and a 2nd image substantially corresponds. FIG. 4B is an explanatory diagram showing the frequency of the first image, and FIG. 5C is an explanatory diagram showing the frequency of the second image. It is explanatory drawing of the method of calculating the brightness | luminance difference for every coordinate range, (a) is explanatory drawing of an attention coordinate and surrounding coordinates, (b) is explanatory drawing of the attention coordinate and surrounding coordinates in the following attention coordinate. . FIG. 4 is an explanatory diagram of pasting of the first image and the second image in the Z-axis direction, (a) is a diagram showing the subject, (b) is an explanatory diagram of the first image, and (c) is the first image. 2D is an explanatory diagram of the second image, FIG. 3D is an explanatory diagram in which a pixel having a large luminance difference is extracted from the second image, and FIG. 2E is an explanatory diagram in a state in which the extracted pixel is pasted on the first image. It is. It is explanatory drawing which pastes a 3rd image to a 1st and 2nd bonded image in a Z-axis direction, (a) is explanatory drawing of a 3rd image, (b) is luminance from a 3rd image. (C) is an explanatory diagram of a state in which the extracted pixels are pasted on the first and second combined images, and (d) is a fourth image of the first image. It is explanatory drawing which shows the state affixed on the bonded image from 1 to 3. FIG. It is explanatory drawing which pastes together the image picked up sequentially, moving to a XY-axis direction, (a) is explanatory drawing of a 3rd image, (b) is the 3rd which shows the state which was cut out with the bonded image. (C) is an explanatory view showing a state where the third image is pasted together, and (d) is an explanatory view for reducing the pasted image according to the display. It is a flowchart explaining the image processing method of this invention. It is a flowchart explaining the moving direction determination procedure,

  Embodiments according to the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing a basic configuration of an image processing apparatus 1 according to an embodiment of the present invention. An image processing unit 2 such as a personal computer, an image acquisition unit 3 such as a camera, a moving unit 4 on which a subject 5 is placed and moved in the XY axis direction or the Z axis direction, and an image display unit 6 such as a display for displaying an image are provided. The moving unit 4 may be a unit that moves the image acquiring unit 3 such as a camera in the XY axis direction or the Z axis direction. The image processing unit 2 includes a moving direction determination unit, an XY-axis direction bonding unit, a Z-axis direction bonding unit, and an image display unit.

  Further, as shown in FIG. 9, the image processing method 10 of the present invention goes through a moving procedure 61, an image acquisition procedure 62, and a moving direction determination procedure 63, and then an XY-axis direction joining procedure 64 or a Z-axis direction joining procedure 65. And an image display procedure 66 is provided.

An image processing apparatus 1 according to the present invention is an image processing apparatus 1 that performs image processing for displaying an image captured by a camera in real time on a display as a panoramic image while relatively moving the subject and the camera. Alternatively, a moving unit 4 that moves the camera in the XY axis direction or the Z-axis direction, and an image acquisition unit 3 that captures images of the subject while moving the moving unit 4 and continuously acquires images.
The frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state in which a plurality of image data is pasted together, and the second acquired after one frame of the first image data The luminance component for each coordinate of the image data is compared with the frequency obtained by frequency analysis, the coordinate range in which the frequency difference for each coordinate is minimized is searched, and the corresponding coordinate range is further determined in the search for each image data The coordinates where the luminance difference between the two images is minimized are searched for, and the searched coordinates are specified as the coordinates of the matching part of the subject in the first image data and the second image data, respectively. If the coordinates of the data and the coordinates of the second image data are different, it is determined that the subject has moved relative to the XY axis, and if the coordinates match, it is determined that the subject has moved relative to the Z axis. Moving direction determining means and the moving direction determining means determine that the subject has relatively moved in the XY axis direction, the first image data and the second image data are identified by the moving direction determining means, respectively. When the XY axis direction pasting means for matching the coordinates in the XY axis direction and sequentially pasting them in time series and the movement direction judging means determine that the subject has moved relative to the Z axis direction, the first image The difference in luminance between each pixel is calculated for each coordinate range consisting of several pixel ranges of each of the data and the second image data, and each difference is compared for each pixel of the same coordinate, Z-axis direction pasting means for pasting together pixels of coordinates having a large difference in time sequence in sequence to the first image data.

If the moving means 4 is a mechanism that moves the stage on which the subject is placed in the XY axis direction or the Z axis direction, or a mechanism that moves the camera in the XY axis direction or the Z axis direction, it can be automatically moved by manual movement. It may be moved.

  The image acquisition unit 3 may be a camera that captures images of a subject while the subject or camera is moved by the moving unit 4 and continuously acquires images. The acquired image may be a color image or a monochrome image, and the range of the captured subject image and the range of the subject image captured after the next frame are photographed so that at least a part of the captured range can be acquired. . Images taken by the camera are sequentially stored in the image processing apparatus 1 as image data in time series.

The moving direction determining means will be described. First, the first image data in the acquired state or in a state in which a plurality of image data is acquired and the second image data acquired after one frame of the first image data are compared. The first image data refers to image data of one captured image or image data that has been sequentially pasted together in time series, and a destination to which image data of the next captured image is pasted. Image data. The second image data is image data to be imaged after one frame and then to be pasted. Both the first and second image data may be color images or monochrome images.

Next, the luminance range for each coordinate of the first image data and the second image data is compared with the frequency obtained by frequency analysis, and a coordinate range in which the frequency difference for each coordinate is minimized is searched. When each image data is a monochrome image, it is not converted as it is as a single-color grayscale image, and when it is a color image, it is converted into single-color grayscale image data.

The search for the coordinate range in which the frequency difference of the luminance component is minimized may be performed using a grayscale image having the size of the captured image, or may be performed using a reduced image obtained by reducing the grayscale image. When the reduced image is used, the search time for the entire image can be shortened.

The search for the coordinate range where the difference in the frequency waveform of the luminance component is minimized is because the waveform of the frequency in the common range is substantially the same if the image is taken with a part of the subject in common. Each coordinate range in which the frequency waveform is substantially the same is searched from the image and the second image.

First, the outline range of the coordinate range in which the first image data and the second image data are both photographed with a part of the subject in common is obtained. The search of the coordinate range at this stage is a rough search. Therefore, the frequency analysis of the luminance of each pixel of the first and second grayscale image data or the reduced image data of the first and second grayscale image data is performed, for example, in the horizontal order or in the vertical order over all the pixels. Do it. Then, the first and second grayscale image data, or the first and second reduced-scale image data are compared in frequency waveform, and the first and second grayscale images substantially match in frequency waveform. Each coordinate range of the data or each coordinate range of the first and second reduced image data is searched and specified. This stage corresponds to “Identify coordinate range with minimum difference by comparing frequency waveforms obtained by frequency analysis of luminance components for each coordinate” 631 in FIG. This stage uses image data 7 or reduced image data 8.

Comparison of frequency waveforms will be described with reference to FIG. FIG. 4B shows the frequency 21 with respect to the coordinates of the first image 11, and FIG. 4C shows the frequency 22 with respect to the coordinates of the second image 12. When the first image 11 and the second image 12 are imaged in the positional relationship as shown in FIG. 4A, the portion where the same part of the subject 5 is imaged is also the second image from the first image 11. A waveform 20 having substantially the same frequency is also shown from the image 12. Therefore, the coordinate range 201 indicating the frequency waveform 20 substantially matches the frequency waveform, and the coordinate range 201 of the first image 11 and the coordinate range 202 of the second image 12 are specified.

In the case of a reduced image, the size in the XY-axis direction is reduced to 1/2 to 1/20, preferably 1/10 to 1/20, compared to before reduction. As a result, although the time for searching for the coordinate range in which the waveform of the frequency is substantially the same varies depending on the image data such as the image size, the search time for one image is approximately 1 for the gray scale image in the experiment of the inventors. The time required for the reduced image could be reduced to about 30 μsec. Since shooting with a camera requires approximately 30 μsec per shooting, it is possible for the user to realize that the bonding is realized sequentially in real time in real time.

Next, an elaborate search is performed with the coordinate range obtained by the search for the first and second image data as a center, including the surrounding coordinate range, and both the first and second image data are searched for the subject. Search for a coordinate range in which some parts are shared. In an elaborate search, the brightness of the common range is substantially the same if the image is taken with some parts of the subject in common, so the coordinates where the brightness is substantially the same from the first image and the second image are obtained. Explore. In this search, the reduced image is not used because it is a rough search.

With respect to the respective coordinate ranges of the specified first and second grayscale image data, or the respective coordinate ranges of the specified first and second reduced image data, each obtained by the search result. A search is made for a coordinate whose luminance substantially matches the corresponding coordinate range. The coordinates where the luminances of the search results substantially match are specified as the coordinates of the matching part of the subject in the first grayscale image data and the second grayscale image data, respectively. By this specification, the coordinates of the first image data and the coordinates of the second image data of the same part of the subject are obtained. This stage corresponds to “Identify coordinates with minimum difference by comparing brightness for each coordinate in the specified coordinate range” 632 in FIG. This stage uses image data 7.

When the coordinates of the first image data and the coordinates of the second image data obtained by the precise search are different, it is determined that the subject has moved relatively in the XY axis direction, and the coordinates of the first image data If the coordinates of the second image data match, it is determined that the subject has moved relative to the Z-axis direction. This step corresponds to “determine movement in the Z-axis direction if the specified coordinates substantially match, and move in the XY-axis direction if the specified coordinates are shifted” 633 in FIG. This stage uses image data 7.

Next, the XY axis direction bonding means will be described. When moved in the XY-axis direction, the coordinates in the XY-axis directions of the first image data and the second image data for the same part of the subject are not the same. Therefore, the first image data and the second image data are made to coincide with each other in the XY-axis directions with the coordinates specified by the moving direction determination means, and the second image data is time-sequentially added to the first image data. Are pasted together. When the image captured at this time is a color image, the color image data are pasted together, and when the image is a monochrome image, the monochrome images are pasted together.

In the unlikely event that the first image and the second image are pasted together by the pasting, the boundary between the first image and the second image is noticeable. Since the first image and the second image do not deviate from each other, the boundary between the first image and the second image is not noticeable at all and can be bonded together.

XY axis direction bonding will be described with reference to FIG. FIG. 2A shows the subject 5a, and FIG. 2B shows the imaging positions of the first image 11 and the second image 12 with respect to the subject 5a. In both the first image 11 and the second image 12, the coordinate C corresponds to a part of the subject 5a that is imaged in common. The waveform of the frequency and the brightness in the coordinate C are substantially the same.

The coordinate c corresponds to the coordinate c a of the first image 11 in FIG. 3A and the coordinate c b of the second image 12 in FIG. Then, it can be seen that the coordinates a of the first image 11 and the coordinates b of the second image 12 are the same coordinates, but the coordinates c and a coordinates b are shifted. Thereby, it can be seen that the second image 12 was taken with respect to the first image 11 while moving in the XY-axis direction. Then, as shown in FIG. 3C, for all the first images 11, a portion where the luminance of the second image 12 substantially matches the luminance of the first image 11 is a boundary that is a luminance mismatch boundary. Cut along lines g and h, and bond together as shown in FIG. A single combined image 13 in which the boundary line is not visible is formed in the combined image.

Next, the third image 14 as shown in FIG. 8A is captured, and the luminance of the coordinate range 14A of the third image 14 and the luminance of the coordinate range 13A of the composite image 13 shown in FIG. Are substantially matched, and the luminance of the coordinate range 14B of the third image 14 and the luminance of the coordinate range 13B of the composite image 13 are substantially matched. Then, as shown in FIG. 8 (b), a range where the luminance coincides with the composite image 13 is cut out from the third image 14, and the composite image 15 is created by pasting as shown in FIG. 8 (c). The pasted image 15 is pasted in a state where the boundary line between the images is not visible. In this way, images can be sequentially pasted in time series.

  Next, the Z-axis direction bonding means will be described. When moved in the Z-axis direction, the coordinates of the first image data and the second image data in the XY-axis direction for the same part of the subject coincide with each other. The image is in a state where the in-focus coordinates are different. Therefore, a process of pasting in-focus pixels from the second image taken later to the first image is performed.

  In each of the first grayscale image and the second grayscale image, for each coordinate range consisting of several pixel ranges, between each pixel between the target pixel at the center of the coordinate range and surrounding pixels that are adjacent to the target pixel The calculation for obtaining the luminance difference is performed for all the pixels of each grayscale image while sequentially shifting one pixel at a time. Then, the respective differences are compared for each pixel having the same coordinate, and the pixels having the larger difference in the second image data are sequentially pasted to the first image data in time series.

  The calculation for obtaining the luminance difference will be described with reference to FIG. In FIG. 5A, the target pixel 31 is surrounded by eight surrounding pixels 32. First, the average of the luminance of the eight surrounding pixels 32 is obtained, and the difference between the obtained average surrounding pixel luminance and the luminance of the target pixel 31 located at the center of the surrounding pixels 32 is obtained. As shown in FIG. 5B, this is performed over all the pixels while shifting the target pixel 31 pixel by pixel.

  Next, Z-axis direction bonding will be described with reference to FIGS. An image obtained by photographing the subject 5b shown in FIG. 6A with the camera which is the image acquisition means 3 is set as a first image 41 as shown in FIG. 6B, and then shown in FIG. 6C. The obtained image is set as a second image 42. Then, it is assumed that the frequency waveform and the luminance at the coordinates d a of the first image 41 and the coordinates d b of the second image 42 substantially coincide. The first image 41 is an entirely unclear image, and the second image 42 is an image in which the uppermost flat portion 51 of the subject 5b is clear and the other range is unclear. In this case, since the luminance difference for each pixel of the planar portion 51 of the second image 12 appears larger than the luminance difference for each pixel of the first image 41, the second difference as shown in FIG. Only the pixel range of the planar portion 51 is cut out from the image 42 and pasted as shown in FIG.

  Next, the camera moves in the Z-axis direction, and as shown in FIG. 7A, the third image 44 is an image in which the image of the disk-shaped portion 52 is clearly displayed, and the coordinate ni c is obtained. It is assumed that the frequency waveform and the luminance in FIG. Then, it is determined that the third image 44 has moved from the composite image 43 in the Z-axis direction. Next, the third image 44 is compared with the composite image 43, and the pixel range of the disc-shaped portion 52 of the third image 44 is larger than the pixel range of the composite image 43, and a difference in luminance appears. As shown in FIG. 7B, the pixel range of the disk-shaped portion 52 is cut out and pasted as shown in FIG. In the same manner, when a clear image is obtained from the outer frame portion 53, the images are bonded to create a combined image 46. The coordinates nia, nib, nic, nid and nie in each image are the same coordinates.

  Next, an image display procedure on the display will be described. The entire image of the images sequentially combined in time series by the XY-axis direction combining procedure or the Z-axis direction combining procedure is displayed to fit the display screen size with the aspect ratio fixed.

  As shown in FIG. 8 (c), in order to prevent the combined image 15 from being larger than the display screen 80 and not displaying the entire image, the screen of the display with the aspect ratio fixed as shown in FIG. 8 (d). The image is reduced and displayed to fit the size of 80.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Image processing means 3 Image acquisition means 4 Moving means 5 Subject 6 Image display means 7 Image data 8 Reduced image data 10 Image processing method 11 First image 12 Second image 13 Composite image 20 Waveform 21 Waveform 22 waveform 30 image 31 pixel of interest 32 surrounding pixel 41 first image 42 second image 43 composite image 44 third image 45 composite image 46 composite image 61 movement procedure 62 image acquisition procedure 63 movement direction determination procedure 64 XY axis direction bonding procedure 65 Z axis direction bonding procedure 66 Image display procedure 80 Display screen

Claims (8)

An image processing apparatus that performs image processing to display images as panoramic images on a display in real time by sequentially pasting images captured by the camera while moving the subject and the camera relatively,
Moving means for moving the placed subject or camera in the XY-axis direction or the Z-axis direction;
Image acquisition means for continuously capturing images by capturing an image of a subject while being moved by the movement means;
The frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state in which a plurality of image data is pasted together, and the second acquired after one frame of the first image data The luminance component for each coordinate of the image data is compared with the frequency obtained by frequency analysis, the coordinate range in which the frequency difference for each coordinate is minimized is searched, and the corresponding coordinate range is further determined in the search for each image data The coordinates where the luminance difference between the two images is minimized are searched for, and the searched coordinates are specified as the coordinates of the matching part of the subject in the first image data and the second image data, respectively. If the coordinates of the data and the coordinates of the second image data are different, it is determined that the subject has moved relative to the XY axis, and if the coordinates match, it is determined that the subject has moved relative to the Z axis. And the moving direction determination means that,
When the movement direction determination unit determines that the subject has moved relative to the XY axis direction, the coordinates specified by the movement direction determination unit between the first image data and the second image data are set in the XY axis direction. XY axial direction bonding means for matching and sequentially bonding in time series,
When the moving direction determining means determines that the subject has moved relatively in the Z-axis direction, the luminance difference between the pixels for each coordinate range consisting of several pixel ranges of the first image data and the second image data. Z is calculated, and the respective differences are compared for each pixel having the same coordinate, and the pixels having the larger difference in the second image data are sequentially pasted to the first image data in time series. An image processing apparatus comprising: an axial direction bonding unit. In the movement direction determining means, searching for a coordinate range in which the difference in frequency of the luminance component is minimized,
The first and second image data are reduced to first and second reduced image data, respectively, and the frequency of the luminance component for each coordinate of the first reduced image data is analyzed, and the second reduced image The image processing apparatus according to claim 1, wherein a luminance range for each coordinate of the data is compared with a frequency obtained by frequency analysis, and a coordinate range in which a difference between the frequencies is minimized is searched for. An image processing method for performing image processing for displaying a panoramic image on a display in real time by sequentially pasting together images captured by a camera while relatively moving a subject and a camera,
A moving step of moving the placed subject or camera in the XY-axis direction or the Z-axis direction;
An image acquisition step of continuously capturing images by capturing an image of a subject while being moved by the moving means;
The frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state in which a plurality of image data is pasted together, and the second acquired after one frame of the first image data The luminance component for each coordinate of the image data is compared with the frequency obtained by frequency analysis, the coordinate range in which the frequency difference for each coordinate is minimized is searched, and the corresponding coordinate range is further determined in the search for each image data The coordinates where the luminance difference between the two images is minimized are searched for, and the searched coordinates are specified as the coordinates of the matching part of the subject in the first image data and the second image data, respectively. If the coordinates of the data and the coordinates of the second image data are different, it is determined that the subject has moved relative to the XY axis, and if the coordinates match, it is determined that the subject has moved relative to the Z axis. The moving direction determination step that,
When the movement direction determination unit determines that the subject has moved relative to the XY axis direction, the coordinates specified by the movement direction determination unit between the first image data and the second image data are set in the XY axis direction. XY axial direction bonding step of matching and sequentially bonding in time series,
When the moving direction determining means determines that the subject has moved relatively in the Z-axis direction, the luminance difference between the pixels for each coordinate range consisting of several pixel ranges of the first image data and the second image data. Z is calculated, and the respective differences are compared for each pixel having the same coordinate, and the pixels having the larger difference in the second image data are sequentially pasted to the first image data in time series. An image processing method comprising: an axial direction bonding step. In the moving direction determining step, searching for a coordinate range in which the difference in frequency of the luminance component is minimized,
The first and second image data are reduced to first and second reduced image data, respectively, and the frequency of the luminance component for each coordinate of the first reduced image data is analyzed, and the second reduced image 4. The image processing method according to claim 3, wherein a luminance range for each coordinate of data is compared with a frequency obtained by frequency analysis, and a coordinate range in which a difference between the frequencies is minimized is searched. An image processing program for performing image processing for displaying images on a display as a panoramic image in real time by sequentially pasting images captured by the camera while moving the subject and the camera relatively,
A moving procedure for moving the placed subject or camera in the XY-axis direction or the Z-axis direction;
An image acquisition procedure in which the subject is imaged while being moved by the moving means and images are continuously acquired;
The frequency obtained by frequency analysis of the luminance component for each coordinate of the first image data in the acquired state or in a state in which a plurality of image data is pasted together, and the second acquired after one frame of the first image data The luminance component for each coordinate of the image data is compared with the frequency obtained by frequency analysis, the coordinate range in which the frequency difference for each coordinate is minimized is searched, and the corresponding coordinate range is further determined in the search for each image data The coordinates where the luminance difference between the two images is minimized are searched for, and the searched coordinates are specified as the coordinates of the matching part of the subject in the first image data and the second image data, respectively. If the coordinates of the data and the coordinates of the second image data are different, it is determined that the subject has moved relative to the XY axis, and if the coordinates match, it is determined that the subject has moved relative to the Z axis. The moving direction judgment procedure that,
When the movement direction determination unit determines that the subject has moved relative to the XY axis direction, the coordinates specified by the movement direction determination unit between the first image data and the second image data are set in the XY axis direction. XY axis direction bonding procedure for matching and sequentially bonding in time series,
When the moving direction determining means determines that the subject has moved relatively in the Z-axis direction, the luminance difference between the pixels for each coordinate range consisting of several pixel ranges of the first image data and the second image data. Z is calculated, and the respective differences are compared for each pixel having the same coordinate, and the pixels having the larger difference in the second image data are sequentially pasted to the first image data in time series. An image processing program comprising: an axial direction bonding procedure. In the moving direction determination procedure, searching for a coordinate range in which the difference in frequency of the luminance component is minimized,
The first and second image data are reduced to first and second reduced image data, respectively, and the frequency of the luminance component for each coordinate of the first reduced image data is analyzed, and the second reduced image 6. The image processing program according to claim 5, wherein a luminance range for each coordinate of data is compared with a frequency obtained by frequency analysis, and a coordinate range in which a difference between the frequencies is minimized is searched.   An image that is displayed so that the entire image of the images sequentially combined in time series by the XY-axis direction combining procedure or the Z-axis direction combining procedure fits the screen size of the display with the aspect ratio fixed. The image processing program according to claim 5, further comprising a display procedure.   A recording medium on which the image processing program according to claim 5 is recorded so as to be readable by a computer.

Images