JP2014003357A - Video processing apparatus and video processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a video processing apparatus capable of providing object video data in a display system using image data packets.SOLUTION: The video processing apparatus comprises: an image data division unit which divides captured image data which is a video data obtained by imaging an object by an imaging unit into plural pieces; a packet generation unit which generates an image data packet having an ID for each piece of the image data; and an output unit which collectively outputs image data packets including the pieces of image data and given an identical ID, thereby outputting divided video data.

Description

  The present invention relates to a video processing apparatus that generates video data divided into object video units based on captured video data and reference video data.

  Conventional video signals are progressive scan type or interlaced scan type, and the video of a display device such as a television or a personal computer is displayed according to the standard of each display device, and the entire screen is updated and displayed at a constant cycle. Was designed to be. For this reason, in the progressive scan type or interlaced scan type video signal, even if only a part of one screen changes, it is necessary to update the video signal of all the screens, and the display update rate is the frame rate. Depends on the rate.

  In virtual reality technology (VR) and augmented reality technology (AR), you can try on virtual clothes and accessories before purchasing them on the mail order by synthesizing the video and computer graphics. There is a system that can be used (Patent Document 1). In order to perform such video composition more complicated and freely, the image needs to be divided into images for each object. For example, in order to generate a video of trying on accessories virtually, it is necessary to divide the videos into “customer to try on”, “accessories”, and “background”. In order to generate a divided image for each object, for example, it is necessary to perform image processing for dividing by performing object detection.

  Further, in order to display a composite video with a progressive scan type or interlaced scan type video signal, a new progressive scan type or interlaced scan type composite video is generated by combining the divided video images for each object. A progressive scanning type or interlaced scanning type video signal includes a plurality of standards, and each object video must be a video signal of the same standard. For this reason, when the standards of each object video are different, it is necessary to perform processing for conversion to the standard of the composite video to be generated.

  On the other hand, apart from the progressive scanning type or interlaced scanning type video signal, a display method using an image data packet in which coordinate information of a drawing start position is designated has been studied (Patent Document 2, Non-Patent Document 1). In addition, it has been proposed to improve efficiency by transmitting and receiving video data composed of a plurality of material video data by selecting a video codec for each material video data (Non-Patent Document 2). In the image data packet method, one video is regarded as being generated by a set of a plurality of object videos, each object video is regarded as a time series of frame images, and each frame image is regarded as a set of image data packets. . Each decomposed image data packet has coordinate information that is a drawing start position. According to this method, it is possible to easily generate composite video with addition / deletion / change of display position / change of display time for each object video, but it is easy to generate object video data from given or captured video data. There was a problem that it could not be provided.

JP 05-336445 A JP 2007-264141 A

"Proposal of a new display method that eliminates the concept of full screen and frame rate", IEICE Technical Report, IEICE, July 2011, vol.111, no.116, DC2011-14, pp.15 -twenty two, "Overview of the MPEG-4 Standard", ISO / IEC JTC1 / SC29 / WG11 N4668 (March 2002), Internet <URL: http://mpeg.chiariglione.org/standards/mpeg-4/mpeg-4.htm>

  The video processing apparatus of the present invention can provide object video data in a display method using image data packets.

  In order to achieve the above object, in the present invention, an image data dividing unit that divides captured image data, which is video data obtained by capturing an object by an imaging unit, into a plurality of image data packets having an ID for each of the divided image data. Provided is a video processing device having a packet generation unit for generating and an output unit for outputting divided video data by outputting together image data packets having the same ID in image data packets having divided image data To do.

  The image data dividing unit further acquires reference image data that is other image data obtained by imaging the object, compares the captured image data with the reference image data, and compares the reference image data for a specific point in the captured image data. The corresponding point is obtained, the difference from the corresponding point is calculated, the image data is divided into a plurality of pieces based on the presence / absence of the corresponding point and the difference information, and the packet generator generates reference image data corresponding to the divided captured image data The image data packet is generated including the difference information.

  Furthermore, it has an imaging part comprised by a stereo camera, and the captured image data may be data of one camera of the stereo camera, and the reference image data may be data of the other camera.

  In addition, the image data storage unit further stores image data, and the image data division unit acquires image data corresponding to image data having an object captured by the captured image data as the reference image data from the image data storage unit. To do.

  The image data dividing unit further acquires distance data indicating a distance between an object on an arbitrary scanning line in the captured image data and the imaging unit, divides the captured image data according to a change in the distance, and the packet generation unit An image data packet is generated including distance range information in the captured image data.

  The image data dividing unit further acquires reference image data that is other image data obtained by imaging the object, and acquires distance data by calculating distance data to the object based on the captured image data and the reference image data. .

  According to the present invention, object video data composed of image data packets can be provided. Therefore, an image data packet in which the display position of the object video is easily changed by changing the drawing start position of the image data packet. It is possible to generate and display composite video using the display method. Since one object video data cut out according to the present invention is a set of image data packets, when the object video data is not connected, it can be easily re-divided into connected parts without dividing the image data packet. It can be handled as separate object video data. It is possible to easily generate and display a composite video based on an image data packet display method in which some object videos are deleted or another object video is added. By changing the drawing start position of each of the plurality of object videos, it is possible to easily generate and display a composite video by the display method of the image data packet in which the plurality of object videos are combined.

  By selecting display / non-display of object video data generated according to the present invention in units of object video data, it is possible to easily generate / display a composite video in which display of unnecessary object video is suppressed. Also, since appropriate image compression can be applied to each object video data generated by the present invention, composite video data composed of a plurality of object video data can obtain a high compression rate.

  Also, since the object video data output by the present invention has difference information or distance information between two video data, the object video data is changed to a position different from the original according to the difference information or distance information. Even when the synthesized video data is generated and displayed, it can be easily generated and displayed.

It is a figure for demonstrating the principle of the video processing apparatus 1 of this invention. 1 is a block diagram of a video processing apparatus 1 in a first embodiment of the present invention. It is a figure explaining an example of the division | segmentation process in 1st Example of this invention. It is a block diagram of the video processing apparatus 1 in the 2nd Example of this invention. It is a figure explaining an example of the division | segmentation process in 2nd Example of this invention. It is a block diagram of the video processing apparatus 1 in the 3rd Example of this invention. It is a figure explaining an example of the division | segmentation process in the 3rd Example of this invention. It is a block diagram which shows an example of the hardware constitutions of the video processing apparatus 1 which concerns on embodiment of this invention. It is a figure which shows the example of an application in the case of reproducing | reconstructing and displaying an image | video by an image data packet system. It is a figure which shows an example of the division | segmentation of the video data which image | photographed the to-be-photographed object in the application example. It is a figure which shows the example of a production | generation of the composite display in the application example of this invention. It is a figure which shows the other application example of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram for explaining the principle of a video processing apparatus 1 according to the present invention.

  The video processing apparatus 1 includes an image data dividing unit 11, a packet generating unit 12, and an output unit 13.

  The image data dividing unit 11 divides image data into a plurality of pieces based on, for example, stereo image processing, background difference processing, distance information, and the like. Details of the division processing will be described later. The image data division unit 11 performs division processing on the input video data for each frame image. As the video data, video data captured by a camera or the like may be directly input, or video data stored in a storage unit such as a hard disk may be input. In addition, the video data referred to in this specification includes not only moving image data but also still image data. The image data dividing unit 11 performs a process of dividing frame image data into a plurality of pieces of input video data. In the image data dividing unit 11, the frame image data is divided into a plurality of divided image data.

  The packet generator 12 generates a set of image data packets for each of the image data divided by the image data divider 11. In the display method using image data packets, video is handled as a collection of object videos, object video is handled as a time series of frame images, and frame images are handled as a collection of image data packets. In one aspect, the image data packet is generated in units of one row (or one scanning line) of pixels in the image. In other words, a plurality of image data packets are generated for one piece of divided video data as many as the number of scanning lines necessary for the video data. All image data packets include image data corresponding to the divided image data, drawing start coordinate information in the display device, number of pixels, and the like, and are transmitted to the display device in a packet format. The drawing start coordinate information is, for example, coordinates (x, y) on the display device at the left end position of the image data packet. Note that the image data packet is not limited to one line unit, and may be generated in units of a plurality of lines and a plurality of scanning lines. In this case, the drawing start coordinate information is, for example, coordinate information of the upper left end position or the lower left end position.

  The output unit 13 outputs the image data packet generated by the packet generation unit 12. The video data is output in the form of an image data packet composed of a plurality of image data packets generated by the packet generator 12 for each divided video data. The output image data packet is sent to a display device, for example, and the display device displays video data based on information included in the packet. The output image data packet is stored in the storage device for each divided video data, and when displayed on the display device, the drawing start coordinate information in the packet is rewritten and transmitted to the display device together with other video data. May be.

  FIG. 2 is a block diagram of the video processing apparatus 1 in the first embodiment of the present invention. The imaging unit 20 captures video data and is, for example, a camera or a video camera. The imaging unit 20 includes two imaging units, and includes a stereo camera and two calibrated cameras. The video data acquired by the imaging unit 20 is two video data obtained by imaging the same thing from different positions.

  The image data dividing unit 21 acquires two pieces of video data imaged by the imaging unit 20, and divides each video data. One video data is taken as captured video data, and the other video data is taken as reference video data. The image data dividing unit 21 searches for a pair of specific points and corresponding points in the reference image data corresponding to each scanning line of the frame image data of the captured video data. As for the method of searching for a specific point, for example, the search is started from one point near the center of the region to which the immediately preceding scanning line corresponds. Arbitrary stereo image processing is adopted as a corresponding point search method. As an example, division processing is performed for each scanning line.

  For example, a distance function between pixel sets between a point on a scanning line including a specific point in captured image data and a point on a scanning line including a corresponding point in reference image data is defined, and the maximum continuous region in which the function value is minimized If the section area is within the range of the predetermined section length and threshold value, a corresponding area is set, and the other continuous areas are set as non-corresponding areas. The difference information indicating the shift of the corresponding area at this time is a vector value from the specific point to the corresponding point. Similarly, another specific point and a corresponding point pair in the reference image data corresponding thereto are searched from the uncorresponding region in the scanning line of the captured image data, and a corresponding region in another correspondence relationship is obtained. . When the search for corresponding points is completed for all points on the scanning line of the captured image data, the region with no correspondence disappears, or the pair of specific points and corresponding points is not found in the region with no correspondence If so, the division process ends.

  As another example, an optical flow may be obtained and the division process may be performed at a point where the optical flow becomes discontinuous. The division process may be performed not only on the captured image data but also on the reference image data.

  The packet generator 22 generates an image data packet for each of the divided image data divided by the image data divider 21. The divided divided image data has difference information indicating the adjacent relationship and the shift of the corresponding area. For example, if a distance function is defined between difference information indicating deviation or between optical flows, and the function value is within the range of a predetermined threshold value, the same object video data, otherwise, Let it be another object video data. Further, image data in a region that does not correspond is handled as one object video data in that region. In this way, the packet generator 22 adds the same ID to the packets determined to be the same object video data, and draw start coordinate information (x, y on the display device of the divided video data). ) Is generated. Further, when the divided video data is “image data of a corresponding area”, a vector value which is difference information indicating a deviation from the corresponding point coordinates is generated. When the reference video data is also divided, “image data of an uncorresponding region” in the reference video data is generated as an image data packet.

  The output unit 23 outputs the image data packet generated by the packet generation unit 22. The output unit is, for example, a communication interface. The output unit 23 outputs the divided image data in which the same ID is added to the image data packet as a single object video data (divided video data).

  FIG. 3 is a diagram for explaining an example of division processing in the first embodiment of the present invention. It is a state that the camera A and the camera B of the imaging unit 20 are imaging two rectangular parallelepipeds. Here, the point P corresponds to a specific point of the captured image data A captured by the camera A and a corresponding point of the reference image data B captured by the camera B, and has the same coordinate position (xth column, yth row). Taken with the eyes). Hereinafter, the dividing process in the y-th row will be described.

  The image data dividing unit 21 searches the correspondence relationship between the scanning line including the specific point corresponding to the point P of the captured image data A and the scanning line including the corresponding point corresponding to the point P of the reference image data B, and corresponds. Find the area. As a result, A1 and B1, which are the background, A3 and B3 including the left cuboid, A5 and B5 which are the background, A7 and B7 including the right cuboid, and A9 and B9 which are the background are obtained as the corresponding regions. In the captured image data, A2, A4, and A6 are data of a set of points where no corresponding points exist in the reference image data B. In the reference image data, B4, B6, and B8 are data of a set of points where no corresponding points exist in the captured image data A. Since A8 and B2 are empty sets, they are not shown in the figure. In this way, the scanning line is divided into a plurality of regions. A6 is a region including the background and the rectangular parallelepiped side, but the background part is a background part hidden in the reference image data B with a rectangular parallelepiped, and there is no set of reference image data corresponding to A6, so there is a corresponding point. Since it is divided as a set data of points not to be processed, the background portion and the side portion portion are not separated. B4 is also an area including the background and the rectangular parallelepiped side surface, but for the same reason, in this example, the background portion and the side surface portion are not separated.

Based on the divided data, the packet generator 22 generates an image data packet including data of the drawing start position (left x coordinate) value of the image data obtained by dividing the y-th row of the captured image data A and the divided video data itself. To do. If there is a corresponding divided image data in the reference image data,
“Divided image data, ID, left coordinate of shooting data (x, y), difference information”
If there is no corresponding divided image data,
"Divided image data, ID, left edge coordinates (x, y) of shooting data, mark of unsupported shooting image data (flag)"
Packets are generated. The “divided image data” in this packet is the divided video data itself in one scanning line.
When the reference image data B is divided, among the divided image data, as the image data packet for the divided image data for which the corresponding point does not exist in the captured image data A,
“Divided image data, ID, left edge coordinates (x, y) of shooting data, mark of unsupported reference image data (flag)”
May be generated. The drawing start position is the x coordinate at the left end. However, the present invention is not limited to this, and may be the x coordinate at the right end, or any coordinate data as long as it is the drawing start position of the image data. .

  Further, when generating the image data packet, the packet generation unit 22 uses the same video if the target divided image data has coordinates adjacent to the image data packet of the previous and subsequent frames and the upper and lower scanning lines. Consider the data and assign the same ID. If the image data packet is not adjacent, a new ID is assigned. In addition, when divided video data having different IDs are adjacent to each other by a new scanning line or a new frame, processing for changing IDs may be performed, or information indicating that these IDs are the same is separately provided. , May be output. In this way, video data divided and packetized for each scanning line has the same ID, and can be recognized as a group of divided video data having a plurality of scanning lines.

In the example of FIG. 3, when a specific packet is shown, the x coordinate of the drawing start position of the divided image data is x11, x12,... X17, x19 for each of A1 to A9, and x21 for each of B1 to B9. Assuming x23,... x28, x29, the following image data packets are respectively output by assigning IDs according to the presence / absence of correspondence and the value of deviation.
`` Divided image data A1, ID1, left end coordinate (x11, y), displacement (0, 0) = (x21-x11,0) ''
`` Divided image data A5, ID1, left end coordinate (x15, y), displacement (0, 0) = (x25-x15, 0) ''
`` Divided image data A9, ID1, left end coordinate (x19, y), displacement (0, 0) = (x29-x19, 0) ''
`` Divided image data A3, ID2, left end coordinate (x13, y), displacement (x23-x13, 0) ''
`` Divided image data A7, ID3, left end coordinate (x17, y), displacement (x27-x17,0) ''
"Divided image data A2, ID4, left end coordinate (x12, y), unsupported shooting data mark"
`` Divided image data A4, ID4, left end coordinate (x14, y), unsupported shooting data mark ''
"Divided image data A6, ID4, left end coordinate (x16, y), unsupported shooting data mark"
The total of the divided image data in these image data packets is one scanning line of one image data of video data captured by the camera A.
Furthermore, when outputting divided video data for reference video data,
“Divided image data B4, ID5, left end coordinate (x24, y), unsupported reference data mark”
"Divided image data B6, ID5, left end coordinate (x26, y), unsupported reference data mark"
“Divided image data B8, ID5, left end coordinate (x28, y), unsupported reference data mark”
Is output. For reference image data that is one image data of video data captured by the camera B, a data packet is output for divided image data that does not have data corresponding to the captured image data among the divided image data. Of these image data packets and the divided image data of the photographed image data, the image data packets of “corresponding divided image data” are combined to form one scanning line of one image data of the video data photographed by the camera B.

  As described above, the ID given to the divided image data is given based on the difference (deviation) from the reference image data, and the same ID is given to data having the same difference. That is, according to the above example, the object ID 1 corresponding to the deviation (0,0), the object ID 2 corresponding to the deviation (x23-x13,0), the object ID 3 corresponding to the deviation (x27-x17,0), the correspondence There are four shooting data stamp object IDs 4. These correspond to the background, the front of the left rectangular parallelepiped, the front of the right rectangular parallelepiped, the side of the two rectangular parallelepipeds, and the background object image captured by the camera A that could not be captured because the camera B has a rectangular parallelepiped. .

  Further, the divided image data with respect to the reference image data is one of the non-corresponding shooting data stamp object ID5, and the side surface of the two rectangular parallelepipeds photographed by the camera B and the camera B that cannot be photographed due to the rectangular parallelepiped in the camera A. Corresponds to the object video of the photographed background.

  If there are N correspondences, one scanning line of each image data of each video data is divided into a maximum of (2N + 1), so that there are N corresponding image data packets and a maximum of uncorresponding image data packets ( N + 1) outputs. When outputting image data packets for reference image data, in addition to this, a maximum of (N + 1) image data packets that cannot be handled are output.

  FIG. 4 is a block diagram of the video processing apparatus 1 in the second embodiment of the present invention. The imaging unit 40a is, for example, a camera or a video camera that captures a thing, and outputs captured image data to the image data dividing unit 41. The video data storage unit 40b is a storage unit for storing video data, such as a hard disk and a memory, for example, and stores shooting data captured in the past by the imaging unit 40a and other shooting data. When the captured image data is input from the imaging unit 40a, the image data dividing unit 41 searches and acquires the video data storage unit 40b for video data serving as reference image data obtained by capturing the same thing. The other processes of the image data dividing unit 41, the packet generating unit 42, and the output unit 43 are the same as those of the image data dividing unit 21, the packet generating unit 22, and the output unit 23, and thus description thereof is omitted.

  FIG. 5 is a diagram for explaining an example of division processing in the second embodiment of the present invention. The camera C of the imaging unit 40a captures two rectangular parallelepipeds, and an image obtained by capturing the background in the absence of the two rectangular parallelepipeds as the reference image data D is acquired from the video data storage unit 40b.

  The image data division unit 41 searches for a correspondence relationship between a certain row which is a scanning line of the captured image data C and a corresponding scanning line row of the reference image data D, and obtains a corresponding region. The captured image data is divided into divided image data “the background portion is shifted (0,0) and there is divided image data corresponding to the reference image data”, and the two rectangular parallelepipeds are “divided corresponding to the reference image data. The image data is divided into divided image data, and is sent to the packet generation unit 42, where each is output as an image data packet.

In the example of FIG. 5, when a specific packet is shown, if the x coordinate of the drawing start position of the divided image data is x1, x2,... X5 for each of C1 to C5, the following image data packets are output respectively. Is done.
`` Divided image data C1, ID1, left end coordinate (x1, y), displacement (0, 0) ''
`` Divided image data C3, ID1, left end coordinate (x3, y), displacement (0, 0) ''
`` Divided image data C5, ID1, left end coordinate (x5, y), displacement (0, 0) ''
"Divided image data C2, ID2, left end coordinates (x2, y), unsupported shooting data mark"
"Divided image data C4, ID2, left end coordinate (x4, y), unsupported shooting data mark"
Is output. The total of these image data packets is one scanning line of one image data of video data photographed by the camera C.

There are two object images by division in the example of FIG. 5, object ID 1 corresponding to the deviation (0, 0) and imaging data mark object ID 2 having no correspondence, and respectively correspond to the background and two rectangular object images. Note that the object ID 1 corresponding to the deviation (0,0) is a connected object video. The non-corresponding shooting data mark object ID2 becomes an unconnected object video, and can be divided into two connected parts as necessary.
In the system using the output of the present invention, by dividing the object video output of the present invention for each packet of the connected portion, it is possible to easily obtain two object videos corresponding to the right cuboid and the left cuboid. .

  FIG. 6 is a block diagram of the video processing apparatus 1 in the third embodiment of the present invention. The imaging unit 60a is a means for imaging things, for example, a camera or a video camera. The video data imaged by the imaging unit 60a is input to the image data dividing unit 61.

  The distance data acquisition unit 60b is a unit that is attached to the imaging unit 60a and measures the distance to an object that exists in the imaging range.For example, stereo vision using parallax or a delay time of reflected light is used. Optical distance sensor, reflective beam sensor that projects a beam and observes it with a slightly shifted sensor, laser type optical distance sensor that uses the phase difference of light, ultrasonic type that uses the delay time of reflected sound A sensor and a moire distance meter using a moire pattern. The distance data acquired by the distance data acquiring unit 60b is output to the image data dividing unit 61.

  The image data division unit 61 associates the frame image data of the video data captured by the imaging unit 60a with the distance data acquired by the distance data acquisition unit 60b in units of pixels. The image data dividing unit 61 obtains a difference in distance data between adjacent pixels in each scanning line.If the value is outside the range of a predetermined threshold condition, the image data dividing unit 61 determines that it is discontinuous, To do. After detecting the division points in this way, the image data division unit 61 divides the captured image data at the division points.

  The packet generating unit 62 generates an image data packet based on the divided image data divided by the image data dividing unit 61. The image data packet includes drawing start position coordinate information of the divided image data, for example, left end coordinates (x, y). Further, the image data packet includes distance range information indicating a change in distance included in the divided image data.

  The output unit 63 outputs the divided image data and the image data packet generated by the packet generation unit 62 corresponding to the divided image data. The output unit 23 collectively outputs the divided image data in which the same ID is added to the image data packet, and outputs it as one object video data.

  FIG. 7 is a diagram for explaining an example of division processing in the third embodiment of the present invention. The camera E of the imaging unit 60a is capturing two cuboids. Here, the point P is one point of the background imaged by the camera E, and the distance to the point P is measured by d1 and the distance data acquisition unit 60b.

  The image data dividing unit 61 sequentially acquires distance data corresponding to points on the scanning line including the specific point P of the captured image data E, and detects a portion where the distance changes discontinuously. For example, it is a part that changes from d1 to d2. The area on the left side of this discontinuous point is E1 and the area on the right side is E2 divided image data. Thereafter, a portion that continuously changes from d2 to d3 and discontinuously changes from d3 to d1 is detected. Let E2 be the detected discontinuity and E3 be the right side. Similarly, let E3 be the discontinuity from d1 to d4, and let E4 be the right. After that, it changes continuously from d4 to d5, and the part where it changes discontinuously from d5 to d1 is E4, and the right is E5. In this way, the image data is divided by the points on the scanning line.

The packet generation unit 62 generates an image data packet including data of a drawing start position (left-end x coordinate) value of image data obtained by dividing the y-th row of the captured image data E based on the divided data. The image data packet is generated including the distance range information included in the divided image data. That is, the generated image data packet is
"Divided image data, ID, left edge coordinates (x, y) of shooting data, distance range information"
Packets are generated.

In the example of FIG. 7, when a specific packet is shown, if the scanning lines at the x coordinate of the drawing start position of the divided image data are x1, x2,. Are output respectively.
"Divided image data E1, ID1, left end coordinate (x1, y), distance range [d1, d1]"
“Divided image data E2, ID2, left end coordinate (x2, y), distance range [d2, d3]”
“Divided image data E3, ID1, left end coordinate (x3, y), distance range [d1, d1]”
"Divided image data E4, ID3, left end coordinate (x4, y), distance range [d4, d5]"
"Divided image data E5, ID1, left end coordinate (x5, y), distance range [d1, d1]"
As described above, the packet generation unit outputs the divided image data, the drawing start position coordinate information, and the distance range information corresponding to the divided image data.

  The object video by the division in the example of FIG. 7 includes an object ID 1 corresponding to the distance range [d1, d1], an object ID 2 corresponding to the distance range [d2, d3], and an object ID 3 corresponding to the distance range [d4, d5]. These correspond to the object image of the background, the left cuboid, and the right cuboid, respectively. The object ID1 corresponding to the distance range [d1, d1] is not connected in the y-th row, but is connected in a row where no cuboid is present, and as a whole becomes a connected object video.

  The division of FIG. 7 is the same division as that of the second embodiment of FIG. 5, but the number of object videos to be output is different from three.

  FIG. 8 is a block diagram showing an example of a hardware configuration of the video processing apparatus 1 according to the embodiment of the present invention. In FIG. 8, the computer constituting the video processing apparatus 1 can be realized by a general-purpose hardware configuration that has existed before. That is, as shown in FIG. 8, the computer forming the video processing apparatus 1 includes the CPU 81, ROM 82, RAM 83, external storage device 84, communication interface 85, camera 87 and sensor 88 connected to the input / output interface 86, and display. A display 89 provided as a device is configured to be connected to a bus. For example, the camera 87 corresponds to the imaging units 20, 40a, and 60a, and the sensor 88 corresponds to 60b. The CPU 81 corresponds to the image data division unit and the packet generation unit. The video data storage unit 40b is, for example, an external storage device 84, and the output unit 43 is a communication interface 85 or an input / output interface 86.

  For example, when displaying the image data packet output from the video processing apparatus 1 according to the embodiment of the present invention on the display 89, the display displays the divided video data, the drawing start position, and the number of pixels included in the image data packet. Based on this, video data composed of a plurality of image data packets is displayed.

FIG. 9 is a diagram illustrating an example of a subject in an application example in which a video is reconstructed and displayed by the image data packet method based on the divided image data generated according to the present invention. Description will be made based on the case where a subject such as 90 is photographed by the video processing apparatus of the present invention, for example, the stereo camera of the first embodiment of the present invention. In 90, a mountain is shown in the distance as a background, and a rectangular parallelepiped object is placed in front.
FIG. 10 is a diagram illustrating an example of division of video data obtained by photographing a subject in an application example. When the displacement of the cuboid subject when shooting with the stereo camera is (x0,0), the object image 91 (FIG. 10 (a)) in the background corresponding to the shift (0,0), the shift (x0,0) ) Corresponding to the object video 92 (FIG. 10B) and the non-corresponding shooting data mark object video 93 (FIG. 10C) are output. When the cuboid is sufficiently far away from the distance between the optical axes between the cameras A and B, the image of the side surface of the cuboid is included in the object image 92 corresponding to the deviation (x0,0). Here, assuming that the rectangular parallelepiped is sufficiently far away, the object video 92 is a rectangular parallelepiped object video.

  When the divided video data with respect to the reference video data is also output, an unsupported reference data mark object video 94 (FIG. 10D) is output. That is, 91, 92, and 93 are the captured image data 90 of the camera A, and an image obtained by combining the object image 91 and the object image 94 with the image obtained by moving the drawing position of the object image 92 to the left by x0 is captured by the camera B. Reference video data.

The object video data, which is the video data divided in this way, has image data packets of at least the number of scanning lines per frame image data in each object video data. For example, an image data packet of a scanning line in the object video 92 is
"Divided image data A3, ID2, left end coordinate (x, y), displacement (x0, 0)"
It is.

  As described above, the rectangular parallelepiped object video 92 is a total of packets to which the same ID (here, ID2) is assigned in a plurality of scanning lines. For example, a control unit such as a CPU of a display system extracts a data packet having the same ID from image data packets stored in a storage unit such as a memory and sends it to a display device, thereby generating a rectangular parallelepiped object video. Can be displayed. In addition, by changing the drawing start position of the image data packet constituting the cuboid object video 92, it is possible to change the display position or generate a composite video in combination with other video.

FIG. 11 is a diagram illustrating a generation example of a composite display in an application example of the present invention. For example, a composite video 96 (FIG. 11B) is obtained by superimposing and displaying a rectangular parallelepiped object video 92 on a separately captured background object video 95 (FIG. 11A). That is, the drawing priority order of the rectangular parallelepiped object video 92 may be set higher than the drawing priority order of the separately photographed background object video 95 and given to the display device. That is, the drawing priority order may be set high in the command packet for the object video 92. For example,
If the drawing priority of the background object video 95 is 10, the drawing priority of the rectangular object video 92 is 11.
By using the command packet, the display device displays a video 96 in which a rectangular object video 92 and a background object video 95 overlap with each other by displaying a rectangular parallelepiped object video 92. Can be synthesized. The command packet is a packet generated separately from the image data packet, and is generated and transmitted for an instruction command to the display device.

  FIG. 12 is a diagram showing another application example of the present invention. 97 synthesizes and displays the object image 92 in a superimposed position in response to the movement of the position of the user wearing the sensor capable of measuring the position (for example, a position sensor such as a three-dimensional magnetic sensor). A display device is shown. By changing the drawing start position of the image data packet in the object video 92 according to the change in the position of the user, the object video 92 can be easily moved and displayed. In FIG. 12, the display device and the sensor attached to the user are connected by wire, but may be wireless.

  The present invention is applicable to a stereo camera, a camera having a distance meter, and the like.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 11 Image data division part 12 Packet generation part 13 Output part

Claims (7)

An image data dividing unit that divides the captured image data, which is video data obtained by imaging the object by the imaging unit, into a plurality of parts;
A packet generator for generating an image data packet having an ID for each of the divided image data;
An output unit that outputs divided video data by collectively outputting image data packets having the same ID in image data packets having divided image data;
A video processing apparatus. The video processing apparatus according to claim 1,
The image data dividing unit further acquires reference image data that is other image data obtained by imaging the object, compares the captured image data with the reference image data, and performs the reference to a specific point in the captured image data. Find the corresponding point of the image data, further calculate the difference with the corresponding point, divide the image data into a plurality based on the presence of the corresponding point and the difference information,
The video processing apparatus, wherein the packet generation unit generates an image data packet including difference information between the divided captured image data and the corresponding reference image data. The video processing apparatus according to claim 2,
In addition, it has an imaging unit composed of a stereo camera,
The video processing apparatus, wherein the captured image data is data of one of the stereo cameras, and the reference image data is data of the other camera. The video processing apparatus according to claim 2,
Furthermore, it has a video data storage unit for storing video data,
The video processing apparatus, wherein the image data dividing unit acquires image data corresponding to image data having an object captured by the captured image data as the reference image data from the video data storage unit. The video processing apparatus according to claim 1,
The image data dividing unit further acquires distance data indicating a distance between an object on an arbitrary scanning line in the captured image data and the imaging unit, and divides the captured image data according to a change in distance,
The video processing apparatus, wherein the packet generation unit generates an image data packet including distance range information in the divided captured image data. The video processing device according to claim 5,
The image data dividing unit further acquires reference image data that is other image data obtained by imaging the object, and calculates distance data to the object based on the captured image data and the reference image data. A video processing apparatus characterized by acquiring distance data. Dividing imaged image data, which is video data obtained by imaging an object, into a plurality of pieces;
Generating an image data packet having an ID for each of the divided image data;
Outputting divided video data by collectively outputting image data packets assigned with the same ID in image data packets having divided captured image data;
A video processing method.

Images