JP2000163596A - Animation processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To directly recognize an image with high precision, where two or more colors coexist, while using a simple pattern matching method. SOLUTION: Two or more discrimination designating colors are simultaneously designated in a mask area MSK which is set on a pixel plane SC so as to be movable, the coexistence of the pixels of the respective discrimination designating colors in the mask area is decided to be necessary and required for a pattern condition and it is judged whether the pixels of the respective discrimination designating colors are adaptive to the pattern condition or not in the mask area which is set on the pixel plane. Thus, pattern matching (adaptation) is executed in a state to simultaneously detecting two or more colors of pixels in the mask area MSK so that the image where two or more colors coexist is directly recognized with high precision. Besides, when the plural numbers, positions and discrimination designating color pixels are included as the condition for discrimination, the condition including at least one of their correlative arrangement relations is used so that processing contents are simplified and also adaptation to a real-time processing is easily attained.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a moving image processing device.

[0002]

2. Description of the Related Art In the field of image processing, there are techniques such as noise removal and image correction by filtering and pattern matching, and image extraction for efficiently extracting only desired image information in a mixture of various image information. Various technologies have been developed. Since the amount of image data to be processed is generally enormous, application to the field of still images has conventionally been considered exclusively in consideration of the processing capability of easily available hardware. However, in recent years, digital image technology has been remarkably developed, and the situation has changed drastically in conjunction with the spread of inexpensive and high-performance computers, and interest in moving image processing technology is rapidly increasing in each field.
Although its application field is extremely wide, it can be applied to, for example, digital broadcasting, computer graphics (CG), motion capture systems, and industrial inspection techniques.

[0003]

However, interest in and demand for moving image processing has grown so rapidly that the accumulation of conventional image processing technology has been mainly performed in the field of still images. It cannot be denied that specific techniques such as image correction and image extraction are still behind in the field of still image processing. In particular, in the case of a full-color image or a multi-tone grayscale image, it is possible to some extent to directly perform pattern matching by comparing a model image with a recognition target image in the field of still images in which a software method is easily used. However, it cannot be said that processing of a moving image including a large number of moving image frames and requiring real-time processing is often impossible unless a large-sized computer or the like is used.

If the moving image data is binary image data, for example, the data of each moving image frame is fetched into a storage device, and a partial binary image pattern of a simple binary image pattern is stored for each frame. A technique for shortening the processing time by repeating matching is also being studied on a software basis. In the case of a color moving image, if it is considered that the image is a binary image focusing only on a specific color, it seems that a similar method can be applied. However, this has the following disadvantages, and its use is also limited. (1) An image in which two or more colors are mixed cannot be directly recognized. It is conceivable to repeat the binary matching process while changing the color of interest, but this not only increases the number of data processing operations but also the relative positional relationship between regions having different colors (for example, the red region and the blue region Information about whether they are always adjacent to each other). (2) Real-time processing while capturing or reproducing moving images is difficult.

[0005] On the other hand, depending on the purpose, for example, it is often the case that it is desired to extract only a desired number of colors from a color image. A specific example is a case where a color marker attached to a subject is detected in a motion capture system. However, in a motion capture system that handles color images, the amount of information of color image data is orders of magnitude larger than that of a grayscale image, so a considerable amount of memory and a high-speed CPU are required, and the system becomes larger. There is a problem that it is difficult to introduce easily because of the high price. In particular, if an attempt is made to introduce a system using a plurality of cameras for color images, the amount of required memory becomes enormous, and the problem of system complexity and high price becomes remarkable, which has become a bottleneck for its widespread use. In addition, the colors of the markers attached to the body of the subject are limited in the number of colors because the markers on the chromaticity diagram must have a certain degree of separation in order to facilitate identification of markers at different positions. As a result, there is a disadvantage that the number of markers is limited to at most about 15 colors.

An object of the present invention is to provide a moving picture processing apparatus capable of directly and accurately recognizing an image in which two or more colors are mixed while using a simple pattern matching technique. To provide a video processing device that can dramatically increase the processing speed by using the configuration and easily cope with real-time processing. When applied to a motion capture system, the number of usable markers can be greatly increased Providing a device.

[0007]

Means for Solving the Problems and Action / Effects In order to solve the above-mentioned problems, a first structure (claim 1) of the moving image processing apparatus according to the present invention comprises a color and / or density (hereinafter, referred to as both). A predetermined shape and size of image data of each frame of a color or grayscale moving image in which three or more types of pixels having different output settings (generally referred to as “color”) are different from each other are mixed. Area setting means for setting the virtual mask area to be movable on the pixel plane of the image data, and determining two or more colors among three or more colors set for each pixel by designating a specified color As the pattern condition, it is essential that the pixels of the respective designated colors coexist in the mask area, and the number of the designated designated color pixels in the mask area, the mask area of the designated designated color pixels Within In the case where a plurality of pixels of the specified color are included, a pixel including at least one of the relative arrangement relations is predetermined, and the pixels of each specified color are set in the mask area set on the pixel plane. Is a pattern matching determining means for determining whether or not the pattern condition is satisfied. One or a plurality of pixels satisfying a predetermined positional relationship with a set mask area are set as target pixels, and a pattern A target pixel data generating unit that generates target pixel data that defines a specific output state according to a result of the determination by the matching determining unit.

In the above-described apparatus configuration, two or more discriminating designated colors are simultaneously designated for a mask area set to be movable on a pixel plane, and a pixel of each discriminating designated color is included in the mask area as a pattern condition. The coexistence is defined as essential, and it is determined whether or not the pixel of each specified color in the mask area set on the pixel plane conforms to the pattern condition. Thereby, pattern matching (adaptation) is performed in such a manner that pixels of two or more colors are simultaneously detected in the mask, so that an image in which two or more colors are mixed can be directly and accurately recognized. In addition, since the discrimination conditions include at least one of the number and position of the discrimination-designated color pixels and, when a plurality of discrimination-designated color pixels are included, their relative arrangement relations, the processing contents Can be simplified, and application to real-time processing can be easily achieved. The number, position, or relative position of pixels may be detected by focusing only on pixels of the same color, or the total number, relative position, or the like of pixels having different colors may be detected.

A second configuration of the moving picture processing apparatus of the present invention (claim 3) integrates and aggregates color information included in each video signal into a specific number of color groups through decoding processing of an original moving picture video signal. Thus, in the form of color-integrated image data having a smaller number of data than the video signal before decoding, three types of output settings of color and / or density (hereinafter, collectively referred to as “color” in a broad sense) different from each other A color integrated image data generating means for generating image data of each frame of a moving image of a color or shade of gradation in which the above pixels are mixed, and a virtual shape having a predetermined shape and size for the image data; A mask area setting means for setting a mask area so as to be movable on a pixel plane of the image data; and determining one or more of three or more colors set for each pixel as a discrimination designated color. In addition, as pattern conditions, the number of discrimination-designated color pixels in the mask area, the position of the discrimination-designated color pixel in the mask area, and, when a plurality of discrimination-designated color pixels are included, their relative arrangement A relationship including at least one of:
A pattern matching judging unit for judging whether or not a pixel of each discrimination designated color conforms to the pattern condition in a mask area set on the pixel plane; and a predetermined positional relationship with the set mask area is satisfied. Target pixel data generation means for generating one or more pixels as target pixels and generating target pixel data for defining a specific output state according to a result of the determination by the pattern matching determination means for the target pixels. And

For example, when the original image data is used as it is in a full-color image or a multi-tone grayscale image, pixels that exactly match the discrimination designation color are rather rare. The number of discrimination designated colors must be extremely increased. On the other hand, it is conceivable to perform a process of calculating the difference between the density or color value between the discrimination designated color and the actual pixel setting value each time and comparing the difference with the threshold value. Large scale and high price,
A reduction in processing speed is unavoidable. However, the second configuration of the present invention can dramatically reduce the amount of information by integrating and aggregating enormous color information of, for example, RGB video signals into a specific number of color groups. Is greatly simplified, and the above problem is solved at once.

Next, the target pixel data generating means generates data for setting the output state of the target pixel to the first output state when it is determined that the arrangement of the pixels of the specified color matches the pattern condition. On the other hand, if it is determined that the pixel is non-conforming, it is possible to generate data for setting the output state of the target pixel to a second output state different from the first output state. This configuration is particularly effective when it is desired to recognize a pixel set that meets the pattern condition as a binary image area.

Further, in the moving picture processing apparatus of the present invention, the mask area scanning means for scanning the mask area on the pixel plane and the target pixel corresponding to the mask area at each scanning position are determined to be suitable for the pattern condition. A suitable pixel extracting means for extracting a pixel (hereinafter, referred to as a suitable pixel) can be provided. This makes it possible to easily extract a region composed of pixels that meet the pattern condition. More specifically, an extracted image data output unit that outputs the data of the extracted suitable pixels as data of an extracted image represented as a set of suitable pixels is provided. If it is desired to extract two or more types of image areas in a form that is distinguished from each other due to a difference in color or a combination of colors, information for specifying each type is added to the matching pixel data. Can be generated by:

An extracted image position data generating means for generating position data of an extracted image represented as a set of suitable pixels based on the position information of the extracted suitable pixels on the pixel plane may be provided. According to this, a specific object (for example, a color marker to be described later) in a moving image is extracted as an extracted image and its position data is generated, so that the movement of the object in the moving image can be easily converted into data. Becomes possible.

One of the most effective applications is a motion capture system. The corresponding moving image processing device specifically has the following requirements. That is, a color camera is provided for photographing the movement of the subject in a state where a color marker is attached to the subject or a part of the subject is defined as the color marker, and the pattern matching determination unit extracts the image information of the color marker. Therefore, one or more colors included in the color marker are set as the discrimination designated colors, and it is decided whether or not the arrangement of pixels of the discrimination designated color in the mask area conforms to the pattern condition. The compatible pixel extracting means extracts at least a part of constituent pixels (hereinafter, also referred to as marker pixels) of a color marker image (hereinafter, referred to as a marker image) as suitable pixels, and extracts extracted image position data. The generation unit is configured to generate a marker image represented as a set of marker pixels based on position information of the extracted marker pixels on a pixel plane. A marker position data generating means for generating position data further includes position data of the marker images generated for each video frame, the motion data output means for outputting the operation data of the subject. This makes it possible to easily create operation data from the position of the color marker attached to the subject.

Next, according to a third configuration of the moving image processing apparatus of the present invention (claim 9), the surface of one color marker is divided into a plurality of areas colored with two or more different marker colors. A color camera for photographing the motion of the subject and a frame of a moving image of the photographed subject in a state where the color marker is attached to the subject or a part of the subject is defined as a color marker. A marker image specifying means for specifying, as a marker image, an area in which two or more specific colors corresponding to the marker colors are present in the image data of (a), a pixel plane of the marker pixels constituting the specified marker image Marker position data generating means for generating position data of a marker image represented as a set of marker pixels based on the position information of The position data of the mosquito image, characterized in that and an operation data output means for outputting the operation data of the subject.

This configuration can also be effectively used as a motion capture system for obtaining motion data of a subject. And as the color of the marker attached to the body of the subject,
The surface of one color marker is divided into a plurality of regions colored by two or more different marker colors, and a region where two or more specific colors corresponding to the marker colors are present is used as a marker. Specified as an image. as a result,
Markers with different color combinations are identified as different markers even if some areas are colored in the same color. Become. For example, when 15 colors are used as identification colors, a maximum of 28 markers can be identified with a combination of two colors, and a maximum of 48 markers can be identified with a combination of three colors.

Also in this case, by integrating and aggregating the color information included in each video signal into a specific number of color groups through the decoding processing of the original moving image video signal, color integration with a smaller number of data than the video signal before decoding is performed. Color integrated image data generating means for generating image data can be provided, and the marker image specifying means can specify a marker image in the color integrated image data. For example, by consolidating and integrating the number of colors to the same number as the set marker colors, a sufficient amount of information necessary for marker identification can be left, and subsequent data can be reduced. It is easy to deal with processing.

In the third configuration, a virtual mask area having a predetermined shape and size is set movably on a pixel plane of the image data for each frame of image data. A mask area setting means for designating one or more colors among a plurality of colors set for each pixel as a discrimination designated color for a marker color; , A position including at least one of the number, the position of the discrimination designated color pixel in the mask area, and the relative arrangement relation between the discrimination designated color pixels when a plurality of discrimination designated color pixels are included, is set on the pixel plane. Pattern matching determining means for determining whether or not the pixels of each specified color in the mask area conform to the pattern condition; and setting the mask area on a pixel plane. A mask region scanning means for scanning,
It is possible to add a suitable pixel extracting means for extracting, from the target pixels corresponding to the mask area at each scanning position, a pixel determined to be suitable for the pattern condition (hereinafter referred to as a suitable pixel). In this case, the marker position data generating means may generate the marker position information based on the position information on the pixel plane of the extracted suitable pixel.
In this configuration, the function of the first configuration is added to the third configuration, and the recognition of the color marker painted in a plurality of colors can be performed with higher efficiency and higher accuracy.

In the above-described moving image processing apparatus of the present invention, the mask area setting means includes a color data storage memory section corresponding to each pixel in the mask area arranged in chronological order of pixel transfer and synchronized with a pixel transfer clock. The memory circuit unit having a transfer control function having a transfer control function of sequentially transferring and storing the color data of each pixel from the upstream side of the array can be configured to the color data storage memory unit. According to this, the setting and movement of the mask on the pixel plane are performed in hardware by pipeline processing synchronized with the pixel transfer clock, so that the processing speed is dramatically improved, Real-time processing while shooting can be easily realized.

In addition, the pattern matching determining means has a color data input terminal corresponding to each pixel in the set mask area, for binaryly inputting whether or not the pixel is set to the specified color for determination. And a hardware logic determination circuit that generates and outputs a binary determination result in a one-to-one correspondence with a combination of the input states of the color data input terminals. According to this configuration, the pattern matching determination unit is configured by hardware logic that does not impose a burden on a control unit configured by a CPU or the like, so that the processing speed is dramatically improved, and the moving image is reproduced or the moving image is captured. Real-time processing and the like can be easily realized. The hardware logic determination circuit is, for example, a logic circuit or a programmable logic device (PL)
D), and furthermore, a look-up table memory or the like.

When two or more types of designated colors are designated, the same number of distinguishing circuits as the designated colors can be provided. In this case, the target pixel data generating means includes a plurality of determination result input terminals to which the determination result from the determination circuit for each determination designation color is binary-input, and one-to-one corresponding to a combination of the input states of the determination result input terminals. In a corresponding manner, the present invention can be configured to include a hardware logic data generation circuit that outputs one of the first output state and the second output state as target pixel data. According to this configuration, the target pixel data generation unit sets C
The processing speed is drastically improved because of hardware logic that does not impose a load on the PU, and real-time processing or the like while reproducing a moving image or photographing a moving image can be easily realized.

Next, a fourth configuration of the moving picture processing apparatus according to the present invention (claim 15) is to provide a moving picture processing apparatus in which two or more types of pixels having different color and / or density output setting states are mixed.
Mask area setting means for setting a virtual mask area of a predetermined shape and size movably on a pixel plane of the image data for each frame of image data; 1 or 2 of the states
In the case where the above is designated as the discrimination designated state, and the pattern condition includes a plurality of discrimination designated state pixels in the mask area, the position of the discrimination designated state pixel in the mask area, and a plurality of discrimination designated state pixels, Are patterns including at least one of their relative arrangement relationships, and a pattern for determining whether or not a pixel in a determined masking state in a determined designation state conforms to a predetermined pattern condition. A matching determination unit, and a specific output state according to a determination result by the pattern matching determination unit with respect to one or a plurality of pixels satisfying a predetermined positional relationship with the set mask area as a target pixel; And target pixel data generating means for generating target pixel data, wherein the mask area setting means corresponds to each pixel in the mask area. The state data storage memory section is arranged in chronological order of pixel transfer, and the output setting state data of each pixel is sequentially transferred to the state data storage memory section from the arrangement upstream side in synchronization with the pixel transfer clock. A memory circuit unit having a transfer control function having a transfer control function for storing, and corresponding to each pixel in the mask area set by the pattern matching determination means, whether or not the pixel is set to a determination designation state Hardware logic discrimination that generates and outputs a binary decision result in a one-to-one correspondence to a combination of the input states of the output setting state data input terminals. It has a circuit.

In this configuration, one or more discriminating designated colors are designated for a mask area set to be movable on a moving image frame, and as a pattern condition, each discriminating designated color in the masked area set on the pixel plane is specified. It is determined whether or not a pixel meets the pattern condition. Further, since the mask area setting means is mainly composed of the memory circuit section with the transfer control function and the hardware logic discriminating circuit, the pattern matching processing in the moving image, which has been conventionally performed on a software basis, is performed. As a result, the processing speed is dramatically improved, and as a result, real-time processing while reproducing a moving image or photographing a moving image can be easily realized.

The moving picture processing apparatus according to the present invention further comprises: an information separating means for separating a color video signal into luminance information as luminance information and color information as color component information through decoding processing; Color information correcting means for correcting the color information based on the luminance information can be provided. In this case, the color integrated image data generating means integrates the corrected color information into a specific number of color groups mainly.
Aggregation makes it possible to generate color integrated image data. Thereby, the system configuration is simplified and the system configuration is less likely to be affected by a change in light amount.

[0025] Also, video signal digital conversion means for digitally converting a color video signal, and image data generation means for generating image data of a subject by taking in the digitally converted color video signal at predetermined time intervals. Can be provided. In this case, the information separation means
With respect to the color information of the pixel of the image data, the color information is divided into luminance information composed of a single component, and a first color component and a second color component.
And color information composed of color components, and the color integrated image data generating means, when the first color component information and the second color component information are divided into a specific number of groups, respectively, Each combination of the information group and the second color component information group and the integrated color information (hereinafter, referred to as
And a color integration look-up table memory storing the correspondence relationship with the integrated color information. For each pixel, the integrated color information corresponding to the first color component and the second color component is stored in the color integration lookup table. It can be configured such that it is read out from the up-table memory in a character-by-character manner and is used as color data of the pixel. According to this configuration, the integration processing of the two color components of the color image data is performed by
This process can be performed quickly using a look-up table (hereinafter, sometimes abbreviated as LUT) memory, and the amount of data involved in the calculation can be greatly reduced. Can be done quickly.

More specifically, the color information correction means, when the integrated color information and the luminance information are divided into a specific number of groups, respectively, combines each of the integrated color information group and the luminance information group, A correction look-up table memory storing the correspondence relationship with the color information after the luminance correction is provided. For each pixel, the corrected color information corresponding to the integrated color information and the luminance information corresponds to the corrected color information of the pixel. The color data may be read from the correction look-up table memory in a text-based manner. In this configuration, the amount of information can be reduced more effectively by performing correction by incorporating the luminance information into the integrated color information by using the correction look-up table memory. Therefore, the accuracy of the generated marker position data also increases.

In the above-described configuration, the two types of color information are integrated, and then the luminance information is renormalized (corrected). However, this order is reversed, that is, each of the color information is corrected by the luminance. Then, they may be configured to be integrated. Specifically, the color information correction means
For each of the first color component and the second color component,
When each of the color information and the luminance information is divided into a specific number of groups, a first and a second storing a correspondence relationship between each combination of the color information group and the luminance information group and the color information after the luminance correction. A correction look-up table memory for each pixel, and for each pixel, the corrected color component information corresponding to each color information and the luminance information is used as the corrected color component information of the pixel as the corresponding correction lookup table. It shall be read out from the table memory. Then, in the color integrated image data generating means, the first color component information and the second color component information each used after the luminance correction are used.

Further, the extracted image position data generating means for generating position data of the extracted image represented as a set of matching pixels, calculates and outputs the extracted image position data based on the designated color pixel data for each image data. It can be configured such that the transfer is performed sequentially in synchronization with the transfer cycle of the pixel (for example, the cycle of the transfer clock pulse of the pixel). According to this configuration, the calculation of the extracted image position data of each image frame is performed at a high speed in a pipeline manner in synchronization with the transfer cycle of the pixels (for example, the frequency of the transfer clock pulse is about 8 to 30 MHz). Therefore, for example, in the case of a motion capture system, it is possible to create operation data including marker position data for each frame in real time during shooting. In particular, 10 MHz (for example, 14 to
By employing a clock pulse frequency of 15 MHz or higher, a high-speed processing environment particularly suitable for, for example, real-time creation of operation data can be realized.

In general, color image data is described in a case where each color component is described as 8-bit (256 steps) data in RGB three-primary color display (the same applies when a luminance component and two kinds of color components are each described in 8 bits). ) ²⁸ ×
2 ⁸ × 2 ⁸ = 16.77 million ways of information volume,
Real-time calculations are almost impossible with the processing power of inexpensive CPUs used in personal computers and the like. However, by reducing the amount of information by integrating and aggregating color information as in the method of the present invention described above, real-time operations can be performed even with a low-cost CPU (computer) having a relatively slow processing speed.

In this case, the extracted image position data generating means generates, for each frame of the image data, the sum Σx of the horizontal coordinate components x of the pixels included in the extracted image area and the sum of the vertical coordinate components y similarly. It can be configured to calculate and output extracted image position data including Σy and the total number n of pixels in each area. According to this, based on the obtained data of Σx, Σy, and n, the center of gravity G of the extracted image (for example, the image of the color marker) is calculated as (Σx / n,
Σy / n). Since the center of gravity is not affected by, for example, the rotational movement of the color marker, it is easy to handle as motion data, and is useful for moving image processing.

Further, Δx and Δy have the advantage that they can be easily calculated, for example, by diverting an image synchronization control unit always provided in a video photographing apparatus system using a normal scanning method. That is, the video signal from the photographing device is decomposed into pixels, separated for each scanning line, and sequentially transferred. At the time of this transfer, the horizontal and vertical synchronization signals are usually inserted into the pixel data string. Therefore, if the pixel transfer clock cycle is constant, the image receiving side receives the horizontal synchronization signal. By counting the number of transfer pixels from, the horizontal coordinate of each pixel on the screen can be specified. Since the transfer pixel count can of course use the pixel transfer clock pulse, it goes without saying that quick processing is possible. On the other hand, by counting the number of scanning lines (e.g., equivalent to the number of horizontal synchronization signals inserted for each scanning line) from the reception of the vertical synchronization signal to the intended pixel position, the vertical coordinates are also calculated. Can be identified. Also, Σ
Since x, Σy and n are all simple addition amounts,
A very simple circuit configuration using an adder enables quick processing, which is advantageous in performing the above-described real-time calculation.

Next, when a lookup table memory is used in the present invention, read target data specifying information such as first and second color component information, luminance information or integrated color information is specified by an address line. By doing so, the read target data specified by the read target data specifying information can be read from the memory cell specified by the address in a text-based manner. Also,
It is possible to provide read control means for sequentially performing read processing of read target data from the look-up table memory for each image data in synchronization with a pixel transfer cycle. By adopting the look-up table memory having the above configuration, the read target data can be quickly accessed by specifying the read target data specifying information by the address line. As a result, high-speed processing synchronized with the pixel transfer cycle is realized, and the above-described real-time processing can be performed without difficulty.

[0033]

FIG. 1 is a block diagram showing an electrical configuration of a moving picture processing apparatus 1 according to one embodiment of the present invention. In this embodiment, the moving image processing apparatus 1 will be described by taking an example in which the moving image processing apparatus 1 is applied to a motion capture system. However, the application field of the present invention is not limited to this. First, the core of the moving image processing device 1 is the operation data creation system 2, which is connected to the host computer 8. The operation data creation system 2 is roughly divided into an image input / output unit (image data generation unit) 4, an image extraction mechanism unit 5, an extracted image data output control unit 6, and a marker pixel counting unit (extracted image position data generation unit, operation data The output unit 7 includes four parts, and is connected via an I / F board 69 to a host computer 8 for creating animations, CG moving images, and the like. The operation data creation system 2 is provided with a clock pulse generation circuit that supplies a pixel transfer clock pulse, and a system controller 3 that manages various controls in the image input / output unit 4.

FIG. 2 is a block diagram showing details of the image input / output unit 4. Subject (for example, a human model such as an actor)
These color cameras 15 for photographing are well-known CCD cameras, and are connected to a video decoder 16. Each video signal (or video composite signal) of R (red signal), G (green signal), and B (blue signal) of the color camera 15 is represented by Y, Cr, and Cv, which are one expression format in which luminance and color are separated. Decoded to a digital signal. In this embodiment, the number of data bits of each signal is set slightly smaller at 6 bits in order to make the concept of the device easier to understand, but the number of data bits of the signal is naturally limited to this. The basic configuration of the device is not changed at all even if a signal other than 6 bits is used (generally, signals of each color are often 8 bits). Is).

Here, since the expression format of the video by Y, Cr, and Cv is known, detailed description is omitted, but Y is a luminance signal, and Cr and Cv are color information of specified chroma and hue. It is a signal, and forms a two-dimensional coordinate space using hue and saturation as independent parameters (corresponding to first and second color component information). By using such an expression format, colors are represented in a two-dimensional space of hue and saturation, making it easy to specify colors. Also, since the luminance is independent, processing that is not easily affected by lighting can be performed. There are advantages.

The decoded Y, Cr, and Cv digital signals are stored in a look-up table memory group 18 (hereinafter, referred to as a look-up table memory group 18).
The look-up table memory may be abbreviated as “LUT”). The LUT group 18 includes two L
UT circuits 19 and 20 are provided. As is known,
The look-up table memory has a function of registering calculation results for all input data in the memory in advance and outputting the calculation results in a subscripted manner by selectively outputting the registered data using an address line for data input. It has.

Generally speaking, the flow of data processing is as follows. Image data is divided and integrated into a limited number of color groups by the color integration LUT circuit 20, and a predetermined type (for example, 64 types if 6 bits, (8 bits, 256 types). Also, the LUT circuit 19 for incorporating a luminance
Corrects the integrated color data with a luminance signal. First, Cr data and Cv data are input to the color integration LUT circuit 20. Then, they are integrated (aggregated) into multiple types of color groups used as color markers,
The luminance is incorporated into the necessary color groups including the luminance in the luminance-incorporating LUT circuit 19. Here, COR (color)
The information is output as 6-bit color group number information of 1 to COR6.

For example, in the case of 8 bits, Cr
Since the component is 8 bits and the Cv component is 8 bits, there are 60,000 thousands of specified color components, which are divided and integrated into 256 types of color groups. Conceptually speaking, 256 necessary color groups from 1 to 256 are set (registered) in advance, and are registered in the luminance-incorporated LUT circuit 19. All combinations of Cr data, Cv data, and the like are sorted (classified) by the LUT circuits 20 and 19 so as to belong to one of these 256 color groups. In other words, the operation results (256 color groups) of all input data (individual 16-bit data) of color data (color or luminance) are registered in the memory, and registered in advance using address lines for data input. By selectively outputting the data (that is, any one of the 256 color groups), the 16-bit color components are integrated into the 8-bit color group. FIG.
Schematically shows an example of a memory cell portion of the LUT circuit, and 16-bit input data is converted to data A
By selecting an address line corresponding to those contents as the data B and the data B, the registration data DR11, D
R2 の is a two-dimensional table to be read.

Note that a static ram (SRAM) is used for the LUT circuits 19 and 20, and 256 kinds of color data can be arbitrarily set. The SRAM does not require a circuit section for refreshing, thus contributing to simplification of the entire system. For example, input image 24
The amount of information can be reduced by integrating and dividing the color representation of about 16.77 million colors (Y: 8 bits, Cr: 8 bits, Cv: 8 bits) for each color camera and outputting 256 types of color group signals. An easy calculation can be realized by reducing the number to about 1 / 60,000.

Each LUT circuit 19 or 20 has the same hardware configuration, which is shown in more detail in FIG.
It is. 3 constitutes one LUT circuit, and corresponds to, for example, the color integration LUT circuit 20 in FIG. In FIG. 3, the input signals of data A and data B correspond to Cr and Cv, respectively, and are output as 6-bit signals of OD. The LUT memory 160 includes an LUT memory controller 159 (hereinafter, LMCN).
In this embodiment, the D-type flip-flop circuits 161 and 162 are provided on the input side of the data.
However, a D-type flip-flop circuit 164 is provided on the output side (hereinafter, the D-type flip-flop circuit is referred to as a D-latch in this specification). Each of these D-latches has an output control terminal (inhibit input terminal) OC.

The data bus from the host computer 8 is connected to the L through a bidirectional bus transceiver buffer 166.
Connected to UT memory 160. The address bus from the host computer 8 is connected to a bus buffer 168,1.
An address for the data A and an address for the data B in the LUT memory 160 are connected to each other via 69.

When initializing the LUT circuit, the host computer 8 sends the LUT memory control signal (LMCNT)
To send a bus request command to the LMCN 159. Accordingly, the LMCN 159 outputs the D latches 161, 162, 1
64 output control signals (OC), the outputs of the D latches 161, 162, 164 are made inactive, and the bidirectional bus transceiver buffer 1
66 and the output enable signals G of the bus buffers 168 and 169 are controlled to make their outputs active. As a result, the host computer 8 becomes readable and writable with respect to the LUT memory 160, and initialization is performed.

After the data input to the LUT memory 160 is completed, the host computer 8 uses the LMCNT to
A bus release instruction (operation execution instruction) is sent to the MCN 159.
Thus, the LMCN 159 controls the output enable signal G of the bidirectional bus transceiver buffer 166 and the bus buffers 168 and 169, and the D latches 161 and 162.
The output control signal (OC) 164 is controlled, and the outputs of the D latches 161, 162, and 164 are activated. As a result, the calculation result obtained by the indexing based on the input data A and the data B to the LUT memory 160 becomes OD.
Will be output to Here, the D latch 161, 1
62 and others use a transfer clock pulse of a pixel to be processed and operate in synchronization with the transfer clock pulse, thereby realizing high-speed arithmetic processing. Note that CK in each D latch is a clock pulse input terminal, and DIR of the bidirectional bus transceiver buffer 166 is a bus direction control signal input terminal. As described above, a pixel transfer clock pulse generated by a pulse generation circuit incorporated in the system controller 3 (FIG. 2) is used as the clock pulse.

In this way, the amount of information can be reduced.
Image data summarized in 6-bit information of ~ COR6
In FIG. 2, after being temporarily stored in the frame memory 21,
It is input to the image extraction unit 5 of FIG. Frame memory 21
Then, the video signal output from the camera output by the interlaced scanning is stored in the odd-numbered scanning memory 22 and the even-numbered scanning memory 23, respectively, and is synthesized to perform scan conversion into a sequential scanning signal. The LUT memory group 18
The control of reading and writing data to and from the frame memory 21 is as follows.
Control signals LTCNT and F from system controller
Performed by MCNT.

Hereinafter, the contents of the processing performed by the image extracting unit 5 will be briefly described, and then, how the processing function is realized by hardware will be described. First, in this apparatus, in order to generate motion data as motion capture, a color marker (hereinafter, also simply referred to as a marker) is attached to a subject such as a human or an animal and photographed. As the color markers, those which are entirely colored in a single color may be used, but in the present embodiment, the surface of one marker is divided into a plurality of regions colored with two or more different marker colors. Use something. FIG. 24 (a) to (o)
Shows various examples of the marker MK formed in a plate shape. As shown in the figure, various shapes of the marker can be adopted, and the coloring pattern is also divided into parallel divisions ((a), (g), (h)), radial divisions ((b),
(C), (d), (f), (i), (k), (l),
(M), (n)), concentric divisions ((e), (j),
Various things such as (p)) are possible. this house,
(A), (b), (e), (f), (k) and (n) are 2
The color markers (c), (g), (j), and (l) are examples of three-color markers, and (h), (m), and (o) are examples of four-color markers.
Further, it can be said that it is desirable to use a marker having a maximum dimension of about 2 to 8 cm (for example, about 5 cm) from the viewpoint of preventing the subject from feeling uncomfortable wearing and ensuring the accuracy of marker recognition.

On the other hand, in the case of a plate-shaped marker, for example, when the marker is photographed from the lateral direction, the marker may enter a blind spot and the marker color cannot be identified. In order to solve such a problem, it is effective to use a three-dimensional shape marker. In this embodiment, since the color on the marker is set as the marker designated color, it is determined that the marker is detected only when all the colors are detected at the same time. This is not desirable because the marker may not be detected. Therefore, it can be said that it is desirable that each color region is formed in a dispersed or intricate manner so that all marker colors can be visually recognized in an arbitrary direction. 25 to 27 show some examples of the three-dimensional marker. For example, in FIG. 25 and FIG. 26, the entire marker is formed in a spherical shape, and in FIG. Has been adopted. (A) shows an example of a two-color marker, (b) shows an example of a three-color marker, and (c) shows an example in which each area is radially divided and further divided into three colors.

FIGS. 26 (a) and 26 (b) show the sphere surface divided by a meridian dividing line V, and FIGS. 26 (c) and (d) divide the spherical surface by a parallel dividing line T. An example in which a color or three colors are applied is shown. (E) shows a background area M,
An example is shown in which an irregularly shaped dispersion region D dispersed therein is painted in three colors. The three-dimensional marker is
Not only a spherical shape but also various shapes can be adopted. For example, FIG. 27 shows an example in which a plurality of angular (or needle-like) protrusions have an outer shape protruding from a spherical main body surface, and the outer surface of each protrusion is painted in three colors.

Such a marker M painted in a plurality of colors
By employing K, the number of markers that can be distinguished from each other can be significantly increased as compared with the case where a single-color marker is used. For example, consider a case where an arbitrary color is selected from the six marker colors (COR1 to COR6) and used as the marker color.
As shown, only six types of markers equal to the number of colors are possible. In the table, “1” indicates a color to be adopted, and “0” indicates a color not to be adopted. MK1 to MK6 each represent a marker number.

[0049]

[Table 1]

[0050] However, in the case of two-color marker will be able to use the same type of marker to the number of combinations for selecting as shown in Table 2, from ₆ 2 (6 _{C 2).} As shown in Table 3, for the three-color marker, ₆ C ₃ = 2
Zero markers can be used. For example, MK in Table 3
# 2 means that the marker surface is painted in three colors of COR1, COR2 and COR4.

[0051]

[Table 2]

[0052]

[Table 3]

The image extracting section 5 (FIG. 1) determines whether or not an image between the colors exists in the photographed image in the following flow. Generate and output extracted image data. That is, as shown in FIG. 12, a mask area (hereinafter, also simply referred to as a mask) MK of a predetermined size and shape is placed on the pixel plane SC of the moving image frame output from the frame memory 21 of FIG. Set to be movable vertically and horizontally on a plane. In particular,
The mask MSK moves horizontally, for example, from the left end to the right end of the scanning line, and when reaching the right end, moves down by one scanning line and returns to the left end, and similarly repeats the horizontal movement on the pixel plane SC. Is scanned.

For example, as shown in FIG. 12A, a case is considered where the mask MSK exists at a certain position on the pixel plane SC. Here, three pixels in the horizontal direction (direction parallel to the scanning line) and three pixels in the vertical direction (direction orthogonal to the scanning line), that is, nine pixels P0 to P9 are arranged in a matrix form as the mask MSK. However, the form of the mask MSK is not limited to this. For example, as shown in FIG. 14, various kinds of pixels such as a pixel arranged in a line in a horizontal direction or a circular shape may be used. Can be.

For example, when a two-color marker composed of COR1 and COR2 (for example, as shown in FIG. 24A) is used, and if the image is taken, the pixel plane as shown in FIG. On the SC, an image area of COR1 (hereinafter referred to as COR1 area) and an image area of COR2 (hereinafter referred to as COR2 area)
Should appear adjacent to each other. For example, when the mask MSK is located so as to extend over the boundary BD between the COR1 region and the COR2 region, the COR
One pixel and the COR2 pixel are detected simultaneously. On the other hand, when the mask MSK is located outside the boundary BD, COR1 and COR2 are not detected at all, or only one of them is detected. However, even if one color is detected, it cannot be determined whether or not it is derived from the image of the marker. This is because there is a possibility that another marker MK uses that color. Therefore, in such a case, the mask MS
K determines that the marker MK has not been detected.
Note that the background portion of the marker MK must be a different color (for example, COR3) from any of the marker colors COR1 and COR2 to be able to be identified as a marker. Therefore, it is obvious that the image data to be used is one in which pixels of at least three colors are mixed.

The simplest way to determine whether the mask MSK has detected the marker MK is to use the mask MSK.
It is sufficient to check whether or not K contains at least one pixel of each marker-specified color. in this case,
If at least one pixel of the designated color exists in the mask, it should satisfy any one of the pattern conditions shown in FIG. In the drawing, hatched pixels indicate pixels of the designated color, and the others indicate pixels of other colors. Therefore, among the color data for each pixel, it is only necessary to sequentially check whether or not the pixel including the color data of the designated color applies to any of the patterns in FIG. 17 for all the colors included in the marker MK. It is. In this check, for example, if the color data of the designated color is binary data, the setting state of the designated color for each of the pixels P0 to P8 is x
It can be easily checked by taking the logical sum of (0) to x (8).

However, if an attempt is made to check only whether or not one pixel of the designated color exists, FIG.
As shown in (a), there is a risk that a marker is erroneously detected under the influence of an isolated pixel PC suddenly generated due to the influence of noise or the like. In this case, for example, it is determined whether at least two or more pixels of the specified color exist in the mask, or as a more severe condition, whether or not a plurality of pixels exist in a form satisfying a specific positional relationship. Try to find out. For example, FIG. 18 shows a pattern condition for checking whether two or more pixels of the designated color are adjacent to any of the upper, lower, left, and right sides. If such a pattern condition is adopted, it is determined that the marker is detected as shown in FIG.
This is a pattern like 9 and, and it is determined that there is no marker. It should be easily understood that by including the diagonally adjacent pixels in the pattern condition, the pattern of FIG. 19 can also be determined as the marker detection.

By performing the above-described pattern matching on the pixel array of the designated color in the mask MSK, if it is determined that a marker has been detected, the mask M
A pixel (target pixel) representing the position of the SK is extracted as a marker pixel (extracted pixel), and data of a marker number that matches the detected combination of the designated colors is output to the address of the pixel. As shown in FIG. 13, the target pixel SPC is defined as the lower left pixel P6 in the mask MSK in the present embodiment, but is not limited to this. If the condition for holding is satisfied, the target pixel SPC is masked MS
It may be set outside K.

For example, when attention is paid to a specific marker number, if the data of the marker number is stored for the address of a certain target pixel on the pixel plane, the output state of the target pixel becomes the first output state. , And if not stored, it can be considered that the output state of the target pixel is the second output state. Then, the target pixel in the first output state is the marker pixel of the marker number.

In the case of the marker MK shown in FIG.
When the mask MSK is scanned on the pixel plane SC while performing the matching under the pattern conditions of FIG. 18 using R1 and COR2 as the marker MK designated colors, the marker pixel becomes as shown in FIG.
It is extracted in the form as shown in (b). These marker pixels are
An extracted marker image is formed by arranging three vertically and two horizontally along the left side of the boundary BD in FIG. This extracted marker image is the boundary B among the images of the marker MK.
This is equivalent to extracting only a partial area along D.

FIG. 22 shows examples of extracted marker images EI for various markers ((a) to (c) are FIGS. 24 (a) to (c), and (d) is FIG. 25. (A) shows a case where each marker is used.) In addition,
G is a marker centroid position calculated from the position coordinates of each pixel of the extracted marker image EI by a method described later.

On the other hand, when the purpose of detecting a marker of a plurality of colors (or use as a motion capture) is deviated and, for example, only one specific color is set as a designated color, a pixel holding a certain positional relationship with the mask MSK. Is set as an extraction pixel when pattern matching is performed, so that the image extraction unit 5 (FIG. 1)
It can be used as a kind of image filter for a designated color (this can be applied to, for example, industrial inspection technology). For example, if the pattern condition of FIG. 18 is adopted, the pattern of “ya” in FIG.
As shown in FIG. 19B, no matter how the mask MSK is aligned, no pattern matching is performed.
As shown in (b), no extracted pixel appears, so that noise is removed. In the other pattern, the same pixel is commonly matched to the mask MSK at a different position, so that the pixel is supplemented and the image is enhanced. For example, a hole opened in the center of the pattern indicates that the extracted pixel EPC appears at the mask MSK position shown in FIG.
It will be filled.

FIGS. 20 (a) to 20 (i) show how the filtering process is performed on the pattern of FIG. 19 (a) by scanning the mask MSK. In each of the diagrams on the right side, the mask MSK is represented by a solid line, the target pixel is indicated by downward-sloping hatching, and the designated color pixel before filtering is indicated by upward-sloping hatching (the target pixel and the designated color pixel before filtering are indicated by hatching). If they overlap, they are both hatched.) Further, in the left-hand drawing, the extracted pixels are represented by hatching falling to the right. In the step (c), since the pattern does not match any of the patterns in FIG. 18, the state of the extracted pixel has not changed from the previous step (b). All other steps are for pattern matching,
It can be seen that the number of corresponding new extracted pixels has increased.

The function of the image extracting unit 5 as described above can be realized by the hardware configuration shown in FIGS. 5 and 7, for example. First, the six color data COR1 to COR6 from the frame memory 21 in FIG. 2 are input to the mask setting processing circuit 30 in FIG. The mask setting processing circuit 30 functions as a mask area setting unit. The data storage memory unit 32 corresponding to each pixel in the mask MSK stores the data in the time series of pixel transfer.
That is, it has a configuration in which the color data (COR1 to COR6: output setting state data) of each pixel is arranged upstream in the data storage memory unit 32 in synchronization with the pixel transfer clock. It operates as a memory circuit unit with a transfer control function having a data transfer control function for sequentially transferring and storing data from the side.

More specifically, a plurality of pixels (pixels arranged in the scanning direction: three pixels P0 to P2, P3 to P5, and P6 to P8 in FIG. ), The data storage memory sections 32 are configured as one type of shift memory circuit (therefore, they themselves function as a memory circuit section with a transfer control function). ). Specifically, a plurality of memory cells 33 corresponding one-to-one to the pixels arranged in the scanning direction are provided, and each time a pixel transfer clock is received, the color data of the corresponding pixel is sequentially stored while being memory-shifted in the transfer time series. It acts as something to do. The color data of each memory cell 33 of each shift memory circuit 32 is used as the color data of the corresponding pixel in the mask as the data output unit 3.
4 is output. By adopting the shift memory, the pixel transfer relative to the mask MSK in the scanning line direction, that is, the relative movement processing of the mask MSK in the scanning line direction is realized in the form of pipeline processing. Thereby, the mask MSK can be moved quickly on the pixel plane,
This is more advantageous for real-time processing.

The individual memory cells 33 of the shift memory circuit 32 are arranged in series in such a manner that the data output terminal is located on the upstream side in the color data input direction, but the data output terminal is connected to the data input terminal on the downstream side. And a D-type flip-flop circuit (D-latch) in which a pixel transfer clock (PCK) is input to each clock input terminal. Then, a data output line 34 as a data output unit is branched from a data output terminal of each D latch. A pipeline processing circuit is realized at low cost by combining the D latches. Reference numeral 31 denotes a D-type flip-flop circuit for input latch of color data.

On the other hand, pixels arranged adjacently at right angles to the scanning direction of the moving image frame (pixels arranged in the scanning orthogonal direction: here, P0, P3, P6, P1, P4, P7, P2, P2 in FIG. 13)
In order to set the mask area in such a manner as to include three mask areas (5 and P8 each), each data storage memory section 32 has a memory area other than the memory area (32 (A)) located on the input topmost side. Line (subsequent memory unit: 32 (B), 32 (C)), which is provided individually for each of the following memory circuits and stores color data of a pixel on a scanning line immediately before a pixel to be stored in each subsequent memory circuit. The memory circuit 35 (LN-1, LN-2) is provided.
These line memory circuits 35 are
The color data of the pixel whose transfer position on the scanning line corresponds to the pixel input to (A) is read from the corresponding memory area, and the corresponding subsequent memory circuit 32 (B), 32 (C) is read.
Forward to By temporarily saving the color data of the pixel one scanning line before in the line memory, the mask setting over a plurality of scanning lines and the moving process thereof can be realized at low cost and at high speed.

In FIG. 5, in order to avoid complication of the drawing, the data output line 34 as a data output unit is used.
Shows only outputs C10 to C18 related to COR1. FIG. 6 shows a corresponding position in the mask MSK of each output.
Is shown in However, actually, the data output lines 34 are also branched from the outputs C20 to C28,..., C60 to C68 of the COR2 to COR6, and the masks MS of the corresponding colors are respectively provided.
It goes without saying that K is formed.

FIG. 15 shows the flow of data in the mask MSK and the line memory circuit 35 (LN-1, LN-2) when focusing on one specific one of COR1 to COR6.
1 shows the data storage state of the data. In (a), 1
This shows a state where the first three pixels of the second scanning line have entered the mask MSK. In this state, the line memory circuit LN-1 of the succeeding memory circuit 32 (B) is
The data of the scanning line is sequentially written. When one new pixel is transferred in this state, the color data of the oldest pixel in the mask is replaced with the color data of the pixel newly entering the mask MSK because the data storage memory unit 32 is a shift memory. The memory shift is performed in an extruded manner, and the mask MSK moves rightward on the pixel plane by one pixel.

FIG. 15B shows a state in which the pixel transfer of the first line is completed, the horizontal synchronization signal is received, and the pixel transfer of the second line is started. As shown in FIG. 5, each line memory circuit 35 receives the horizontal synchronizing signal HSNC as a reset signal and outputs the horizontal synchronizing signal HSN.
When a pixel of a new line is transferred in response to C, the storage address of the pixel is reset and, as in the line memory circuit LN-1 in FIG. Data is written. Further, as shown in FIG. 5, the data of the line memory circuit LN-1 is serially transferred to the line memory circuit LN-2 and copied. As a result, the line memory circuit LN-2 stores data two lines before as seen from the top memory circuit 32 (A). Then, as shown in FIG. 15C, every time the pixel transfer of the second line advances by one, the corresponding data of the previous line is read out from the line memory circuit LN-1, and is read out from the subsequent memory circuit 32 (B ).

FIG. 15D shows a state in which the pixel transfer of the third line has been started. Data transfer from the line memory circuit LN-1 to the subsequent memory circuit 32 (B) is performed as described above. Data transfer from the memory circuit LN-2 to the subsequent memory circuit 32 (C) is performed. FIG. 16E shows a state in which the data of the first three pixels on the third line have entered the mask MSK. As shown in FIG. 16A, the upper left nine pixels on the pixel plane SC have the mask MSK. Corresponding to the shape covered by In this state, each time one pixel is transferred, the mask MSK moves one pixel at a time in the scanning direction. When the mask MSK reaches the right end and receives the horizontal synchronizing signal, it returns to the left end at a position one line lower, Start scanning again. When the frame reaches the lower end of the screen and receives the vertical synchronization signal (VSNC), the frame is cleared, and the same process for the next frame proceeds in real time in synchronization with the moving image frame switching.

The color data of the mask MSK for each color (the number of bits corresponding to the number of pixels in the mask: 9 bits each) is transmitted from the mask setting processing circuit 30 via the data output line 34 to the pattern shown in FIG. It is input to a hardware logic determining circuit 41 (pattern matching determining means) in the matching determining / outputting circuit 40. The hardware logic determination circuits 41 are provided by the number corresponding to the types of designated colors (COR1 to COR6), and the mask setting processing circuit 30
, And has a data input terminal 44 (here, 9 inputs) into which the color data of each pixel is binary-inputted. It generates and outputs results.

The hardware logic discriminating circuit 41 can be specifically constituted by a logic circuit. For example, when the pattern condition shown in FIG. 17 is used, the pattern condition can be easily constituted by one 9-input OR circuit. On the other hand, as shown in FIG. 18, when it is determined whether or not the colors of two specific pixels are set as the discrimination designated colors, as shown in FIG. 9-input AND circuit 47 in which the input of the pixel to be formed and the input of the other pixel are mutually inverted.
Can be exemplified by the types of pattern conditions to be examined. Such a logic circuit may be realized by, for example, a programmable logic device (PLD).
Also, a plurality of discriminating circuits corresponding to each pattern condition are incorporated so that any one of the plurality of pattern conditions can be selected and used, and selector inputs (here, 2-bit inputs of SL1 and SL2) are provided. It is convenient if it can be switched by. In this embodiment, the selector input to each hardware logic determination circuit 41 is provided by the host computer 8 through the register 43.

On the other hand, the hardware logic determination circuit 41
As shown in FIG. 8B, all combinations P1 to PN of the color setting state of each pixel (here, 9 bits, N =
LUT circuit 48 storing the output OP corresponding to (512)
May be used. Here, a two-dimensional LUT circuit is used so that the pattern condition can be switched for each selector input. In this case, the input (and the selector input) of the color data of each pixel from the mask setting processing circuit 30 is LU
An address line is input to the T circuit 48.

Returning to FIG. 7, the judgment result output from each hardware logic judgment circuit 41 (that is, each of the colors COR1 to CORC)
The pattern matching result of OR6) is input to the hardware logic data generation circuit 42. The hardware logic data generation circuit 42 receives the discrimination result from the discrimination circuit of each discrimination designated color as a binary value, and outputs the first output state in a form corresponding to the combination of the input states of the discrimination result input terminals on a one-to-one basis. And the second output state is output as target pixel data. Here, in the case of marker detection, the data of the marker number is output as the first output state, and in the case of not detecting the marker, an output corresponding to none of the marker numbers is output as the second output state. The hardware logic data generation circuit 42 is composed of a decoder of 6-bit input and 5-bit output, and the setting of the marker designation color is selected by the selector inputs SL1 and SL2 (that is, setting of up to four color markers is possible. ), COR1-C
Marker number data is output based on the determination result output of OR6. For example, in Table 2, when COR1 and COR2 are selected as the marker designation colors by the selector inputs SL1 and SL2, the marker number 1 is output when the input for discriminating between COR1 and COR2 is active. In this embodiment, among the COR1 to COR6, the COR
One is prioritized, and erroneous encoding due to multiple detection is suppressed.

The marker number data thus generated is sent to the extracted image data output control unit 6 and the marker pixel counting unit 7 in FIG. 1 as the data of the extracted image (marker image). As shown in FIG. 9, the marker pixel counting unit 7 includes three adders 55 for calculating the sum Σx of the x coordinate values of the pixels of the designated marker number, the sum Σy of the same y coordinate values, and the number of pixels n. ~ 57. The output total of each adder is stored in the data memory 58, and the result is output to the host computer 8. In the data memory 58,
Σx-memory, Σy memory and n-memory are formed.

The $ x-memory is cleared to zero in the initial state. The sub-controller 54 has a built-in address controller that specifies the address of the storage area of the $ x data for each marker color based on the address information from the host computer 8, and stores the memory area of the corresponding address according to the color of the pixel. Select When the first processed pixel comes in, the marker number data of that pixel is input to the address line of the $ x-memory via the address controller.

Then, based on the horizontal synchronizing signal generated by the image transfer synchronizing control unit (built-in to the system controller 3 in FIG. 2), the x counter 52 operates until the target pixel is transferred. Is transferred to the adder 55 as a horizontal coordinate value of the pixel on the screen. The transfer control of the pixels is controlled according to a clock pulse signal generated by a clock pulse generator in the system controller 3 of FIG. On the other hand, the currently stored value of the Σx-memory is fed back to the adder 55, and the count value (ie, the x coordinate value of the new pixel) from the x counter 52 and the cumulative addition value of the Σx-memory are added to the adder 55. 55, and is written again to the same address of the $ x-memory through a buffer or the like (not shown) to update the added storage value.
The processing of the pixel ends.

Next, an adder 56 for calculating Σy of a pixel is paired with Σy-memory. The operation is substantially the same as that of the arithmetic system of が x, except that the input to the adder 56 is from the y-coordinate value of the pixel, that is, from the reception of the vertical synchronization signal from the synchronization control unit to the target pixel position. Is the count output value of the y counter 53 for counting the number of scanning lines (for example, the number of horizontal synchronization signals inserted for each scanning line), and the added output value is the cumulative addition value of the y coordinate. is there.

The adder 57 for calculating n is
It forms a pair with the n memory that stores the output total. The operation is
The operation system is substantially the same as that of the calculation system of Σx, but what is input to the adder 57 is the output of a pixel counter (not shown) that emits one count pulse each time one pixel of the designated marker number enters. The addition output value is the total number n of pixels of the image area (marker image) of the corresponding color marker.

When this series of operations is performed on the entire screen of the color camera 15 (FIG. 2) that is currently in operation, the total Σx of the x coordinate of each color marker (CM) is stored in Σx-memory,
Similarly, a state is obtained in which the sum Σy of the y-coordinates is stored in the 、 y-memory and the total number of pixels n is stored in the n-memory as a calculation result. The stored values are transferred to the host computer 8, respectively. When the processing of the image data of one frame is completed, the contents of each memory are cleared by the signal from the frame counter, and the same processing is repeated for the image data of the next frame. In this way, the values of Δx, Δy, and n of each color marker are calculated and generated in real time as marker position data with respect to the image data sequentially generated by photographing with the color camera 15, and output to the host computer 82. It will be.

For example, as shown in the flowchart of FIG. 11, the host computer 8 of FIG. 1 outputs the number (address) of the color marker designated in S1, and
At step S4, the values of the color markers Σx, Σy and n are read. Then, by using this, as shown in FIG. 22, for example, the barycentric coordinates G of the extracted marker image can be calculated, and this can be obtained as motion data of the subject. This motion data can be incorporated as motion data of the moving image character into CG data such as an animation stored in the CG data storage unit 12 by the host computer 8, for example, and the demo data can be output on the monitor 10. Can be.

On the other hand, the extracted image data output control unit 6 accumulates the data of the marker number in the frame memory 21 in a form associated with the pixel address to make the marker image data, and according to the request from the host computer 8 side, Transfer to this. With the marker image data, the host computer 8 can perform, for example, shape analysis of the marker image.

When using markers that are painted in three or more colors, it is possible to ignore some of the marker colors and apparently recognize them as markers painted in fewer colors. Such processing can be easily performed only by reducing the number of designated colors set in the hardware logic data generation circuit 42 in FIG. FIG. 23 shows an application example of such processing. That is, the marker MK shown in (a) is originally three of COR1 to COR3.
They are painted in different colors, of which COR1 and C
If only OR2 is adopted, it is determined that the area of COR3 does not belong to the marker MK.
Appears near the boundary between COR1 and COR2, and the barycentric position G1 is also calculated as that in the marker image EI1. On the other hand, CO
If only R2 and COR3 are adopted, the area of COR1 is the marker M
Since it is determined that the image does not belong to K, the marker image E
I2 appears near the boundary between COR2 and COR3, and the center of gravity G
2 is calculated as that in the marker image EI2. Since these barycentric positions G1 and G2 have different coordinate values in the original marker MK, for example, G1 on the image
The attitude of the marker MK with respect to the camera can be detected from the values of 1, G2. For example, FIG. 23B shows an example in which the inclination angle θ of the marker MK with respect to the horizontal line X is obtained from the inclination angle of the line segment G1G2 with respect to the horizontal line X, and FIG.
Shows an example in which the inclination angle φ in the front-rear direction with respect to the vertical plane z (coincident with the paper surface) is obtained from the length of the line segment G1G2.

In the moving picture processing apparatus 1, as shown in FIG. 10, a plurality of color cameras 15 are connected to a video switch 3 which is operated in accordance with a command from the host computer 8, and a predetermined color on a subject in one camera 15 is determined. When the marker enters the blind spot, the camera may be appropriately switched to the camera 15 not located in the blind spot.

As described above, the field of application of the moving image processing apparatus of the present invention is not limited to the motion capture system. It can also be applied to devices and the like. Further, the target moving image is not limited to a color image, and can be applied to a grayscale or binary moving image.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall electrical configuration of a moving image processing apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram showing the contents of the image input / output unit.

FIG. 3 is a circuit diagram showing a specific configuration example of a look-up table memory circuit.

FIG. 4 is a diagram conceptually showing data contents of a two-dimensional lookup table.

FIG. 5 is a circuit diagram showing an example of a mask setting processing circuit of the image extracting unit.

FIG. 6 is an explanatory diagram showing a correspondence relationship between a position of a data output line and a pixel position in a mask area.

FIG. 7 is a circuit diagram showing an example of a pattern matching determination / output circuit of the image extraction unit.

FIG. 8 is a diagram illustrating an example in which a hardware logic determination circuit is configured using a gate or a lookup table memory.

FIG. 9 is a circuit diagram showing a configuration example of an extracted image data output control unit and a marker pixel counting unit.

FIG. 10 is a block diagram illustrating an example of a switching circuit of a plurality of cameras.

FIG. 11 is a flowchart illustrating an example of calculating the center of gravity of a color marker.

FIG. 12 is an explanatory diagram showing an example of a mask area setting state on a pixel plane and a marker extraction state based on the state.

FIG. 13 is an explanatory diagram of a mask region.

FIG. 14 is an explanatory diagram showing another example of a mask region.

FIG. 15 is an explanatory diagram of the operation of a mask area and a line memory.

FIG. 16 is an explanatory diagram showing a state of scanning a mask area on a pixel plane.

FIG. 17 is an explanatory diagram showing some examples of pattern conditions when performing one-pixel detection.

FIG. 18 is an explanatory diagram showing some examples of pattern conditions when detecting two adjacent pixels.

FIG. 19 is an explanatory diagram showing some examples of moving image filter processing by pattern matching.

FIG. 20 is an explanatory diagram showing an example of the processing flow.

FIG. 21 is an explanatory view showing another example.

FIG. 22 is an explanatory diagram showing various examples of marker extraction.

FIG. 23 is an explanatory diagram showing a method of recognizing a marker of three or more colors as a marker of a smaller number of colors, and some usage examples thereof.

FIG. 24 is a front view showing various examples of a plate-shaped marker.

FIG. 25 is a front view showing various examples of a three-dimensional marker.

FIG. 26 is a front view showing some other examples.

FIG. 27 is a front view showing still another example.

[Explanation of symbols]

Reference Signs List 1 moving image processing device 2 operation data creation system 3 system controller 4 image input / output control unit (color integrated image data generation unit) 5 image extraction unit 6 extracted image data output control unit 7 marker pixel counting unit (extraction image position data generation unit,
8 Host computer 9 I / F board 10 Monitor 11 Microprocessor 12 CG data storage unit 15 Color camera 16 Video decoder 18 Lookup table memory circuit 30 Mask setting processing circuit (mask area setting means, mask area scanning means) 32) data storage memory section (color data storage memory section, state data storage memory section, memory circuit section with transfer control function) 33 D-type flip-flop circuit (memory cell) 34 data output line (data output section) 35 line memory circuit 40 pattern matching judgment / output circuit 41 hardware logic judgment circuit (pattern matching judgment means) 42 hardware logic data generation circuit (target pixel data generation means, matching pixel extraction means, extracted pixel data output means) 44 color data input terminal 45 discrimination result input terminal 47 gate 48 look-up table memory 52 x counter 53 y counter 55 adder 56 adder 57 adder 58 data memory MSK mask area SC pixel plane SPC target pixel MK color marker

Claims (18)

[Claims] An output setting of a color and / or a density (hereinafter, collectively referred to as “color” in a broad sense) is different from each other.
A virtual mask area of a predetermined shape and size is moved on the pixel plane of the image data for the image data of each frame of the moving image of the color or the gray scale in which more than two types of pixels are mixed. Mask area setting means for setting possible, and two or more of three or more types of colors set for each pixel.
The above items are simultaneously designated as the discrimination designated colors, and the pattern condition requires that pixels of each discrimination designated color coexist in the mask area, and the number of discrimination designated color pixels in the mask area. The position including at least one of the position of the discrimination-designated color pixel in the mask area, and the relative positional relationship between the discrimination-designated color pixels when a plurality of discrimination-designated color pixels are included, is set in advance on the pixel plane. Pattern matching determining means for determining whether or not a pixel of each determination designated color in the mask area matches the pattern condition; and one or more pixels satisfying a predetermined positional relationship with the set mask area. Is used as a target pixel, and target pixel data that defines a specific output state corresponding to the result of the determination by the pattern matching determination unit with respect to the target pixel is generated. A moving image processing device comprising: elephant pixel data generating means. 2. A color integrated image data having a smaller number of data than a video signal before decoding by integrating and aggregating color information included in each video signal into a specific number of color groups through decoding processing of an original moving image video signal. The moving image processing apparatus according to claim 1, further comprising: a color integrated image data generating unit configured to generate a color integrated image data, wherein the pattern matching determining unit determines whether or not the pattern condition is satisfied with respect to pixels on the color integrated image data. 3. A color integrated image data having a smaller number of data than a video signal before decoding by integrating and aggregating color information included in each video signal into a specific number of color groups through decoding processing of an original moving image video signal. In the form of colors and / or
Or density (hereinafter, both are collectively referred to as "color" in a broad sense)
A color integrated image data generating means for generating image data of each frame of a color or grayscale moving image in which three or more types of pixels having different output settings from each other are mixed; A mask area setting means for setting a virtual mask area having a shape and a size so as to be movable on a pixel plane of the image data; and one of three or more types of colors set for each pixel.
Alternatively, two or more colors are designated as the discrimination designated colors, and the pattern condition includes a plurality of discrimination designated color pixels in the mask area, positions of the discrimination designated color pixels in the mask area, and a plurality of discrimination designated color pixels. In this case, it is determined in advance that at least one of the relative arrangement relations is determined, and whether or not the pixels of the respective designated colors match the pattern condition in the mask area set on the pixel plane. Pattern matching determining means for determining whether or not one or more pixels satisfying a predetermined positional relationship with the set mask area are set as target pixels, and the target pixel is determined based on a result of the determination by the pattern matching determining means. A moving image processing device, comprising: target pixel data generating means for generating target pixel data that defines a specific output state. 4. The target pixel data generating means sets an output state of the target pixel to a first output state when it is determined that the arrangement of the pixels of the specified color matches the pattern condition. 2. The method according to claim 1, further comprising: generating data for setting an output state of the target pixel to a second output state different from the first output state when the data is determined to be non-conforming. 4. The moving image processing device according to any one of items 1 to 3. 5. A mask area scanning unit that scans the mask area on the pixel plane, and a pixel determined to be suitable for the pattern condition among target pixels corresponding to the mask area at each scanning position (hereinafter referred to as an adaptive pixel). 5. The moving image processing apparatus according to claim 1, further comprising: a suitable pixel extracting unit that extracts a pixel. 6. The moving image processing apparatus according to claim 5, further comprising an extracted image data output unit that outputs the extracted data of the relevant pixels as data of an extracted image represented as a set of the relevant pixels. 7. An extracted image position data generating means for generating position data of an extracted image represented as a set of the suitable pixels based on the position information of the extracted suitable pixels on the pixel plane. 7. The moving image processing device according to 5 or 6. 8. A method for attaching a color marker to a subject,
Or, in a state in which a part of the subject is defined as a color marker, a color camera for photographing the movement of the subject is provided, and the pattern matching determination unit includes a color camera for extracting image information of the color marker. 1 included
Or, as two or more colors as the discrimination designated colors, it is determined whether or not the arrangement of pixels of the discrimination designated colors in the mask area conforms to the pattern condition. Extracts at least a part of constituent pixels (hereinafter, referred to as marker pixels) of a color marker image (hereinafter, referred to as a marker image) as the compatible pixels. The extracted image position data generating means, Marker position data generating means for generating position data of a marker image represented as a set of the marker pixels based on the position information of the marker pixels on the pixel plane, and further generated for each moving image frame. 8. An operation data output means for outputting the position data of the marker image as operation data of the subject. The placing of the moving image processing apparatus. 9. A method in which the surface of one color marker is divided into a plurality of regions colored by two or more different marker colors, and the color marker is attached to a subject,
Or, in a state in which a part of the subject is defined as a color marker, a color camera for photographing the motion of the subject, and specifying the color corresponding to the marker color in image data of each frame of the moving image of the photographed subject. A marker image specifying means for specifying, as a marker image, an area in which two or more colors coexist, and the marker pixel based on position information on the pixel plane of the marker pixel constituting the specified marker image Marker position data generating means for generating position data of a marker image represented as a set of motion image data; and motion data output means for outputting the position data of the marker image generated for each moving image frame as motion data of the subject. A moving image processing apparatus, comprising: 10. Color integrated image data having a smaller number of data than a video signal before decoding by integrating and aggregating color information included in each video signal into a specific number of color groups through decoding processing of an original moving image video signal. The moving image processing apparatus according to claim 9, further comprising a color integrated image data generating unit that generates the marker image, wherein the marker image specifying unit specifies the marker image in the color integrated image data. 11. A mask area setting means for setting a virtual mask area having a predetermined shape and size to be movable on a pixel plane of the image data for image data of each frame; 1 or 2 among a plurality of colors set for the pixel
The above are designated as the discrimination designated colors for the marker color, and the number of discrimination designated color pixels in the mask area, the position of the discrimination designated color pixels in the mask area, and the plurality of discrimination designated color pixels are specified as the pattern conditions. In the case where it is included, a pixel that includes at least one of their relative arrangement relations is determined in advance, and in the mask area set on the pixel plane, the pixel of each discrimination designated color matches the pattern condition. Pattern matching determining means for determining whether or not there is a mask area; mask area scanning means for scanning the mask area on the pixel plane; and determining that the target pixel corresponding to the mask area at each scanning position matches the pattern condition. Pixel extraction means for extracting the extracted pixels (hereinafter referred to as “adapted pixels”). Video processing apparatus according to claim 9 or 10 and generates the marker position information based on position information on the pixel plane of the adaptation pixel. 12. The mask area setting means, wherein color data storage memory sections corresponding to respective pixels in the mask area are arranged in chronological order of pixel transfer, and the color data storage memory sections are synchronized with a pixel transfer clock. For the storage memory part,
12. The moving image processing apparatus according to claim 1, further comprising a memory circuit unit having a transfer control function having a transfer control function for sequentially transferring and storing the color data of each pixel from the upstream side of the array. 13. A color data input unit for binaryly inputting, for each pixel in the set mask area, whether or not the pixel is set to the specified color, in accordance with each pixel in the set mask area. 13. A hardware logic discriminating circuit comprising a terminal, and a hardware logic discriminating circuit for generating and outputting a binary discrimination result in a form corresponding to a combination of the input states of the color data input terminals on a one-to-one basis. A moving image processing apparatus according to any one of the above. 14. The discrimination designating color is designated by two or more kinds, and the discriminating circuits are provided in a number equal to the discriminating designated color. A plurality of determination result input terminals into which a determination result from the determination circuit of the determination designation color is binary-inputted, and the first output state is provided in a form corresponding to a combination of input states of the determination result input terminals on a one-to-one basis. The moving image processing device according to any one of claims 4 to 8, and 11 to 13, further comprising a hardware logic data generation circuit that outputs any one of a second output state and a second output state as the target pixel data. 15. A virtual mask having a predetermined shape and size for image data of each frame of a moving image in which two or more types of pixels having different color and / or density output setting states are mixed. Mask area setting means for setting an area to be movable on a pixel plane of the image data; and specifying one or more of the output setting states that can be set for the pixel as a discrimination designation state; At least one of the number of the pixels designated in the masking area in the mask area, the position of the pixel designated in the masking area in the mask area, and the relative positional relationship between a plurality of pixels designated in the masking state when the pattern includes a plurality of pixels. Are determined in advance, and the pixels in the discrimination designation state in the set mask area satisfy the predetermined pattern condition. Pattern matching determining means for determining whether or not one or more pixels satisfying a predetermined positional relationship with the set mask region are set as target pixels, and the determination result of the pattern matching determining means for the target pixel is A target pixel data generating unit that generates target pixel data that defines a specific output state in accordance with the specified output state. The mask area setting unit includes a state data storage memory unit corresponding to each pixel in the mask area, and And a transfer control function for sequentially transferring and storing the output setting state data of each pixel from the upstream side of the arrangement to the state data storage memory section in synchronization with the pixel transfer clock. A memory circuit unit having a transfer control function, wherein the pattern matching determination unit determines that each pixel in the set mask area is It has an output setting state data input terminal for binary input of whether or not it is set to the discrimination designation state, and performs binary judgment in a form corresponding to a combination of the input states of the output setting state data input terminals on a one-to-one basis. A moving image processing device having a hardware logic determination circuit for generating and outputting a result. 16. In order to set the mask area so as to include a plurality of pixels (hereinafter, referred to as scanning direction array pixels) that are arranged adjacently in parallel to the scanning direction of the moving image frame,
The memory circuit unit with the transfer control function has a plurality of memory cells corresponding one-to-one to the pixels arranged in the scanning direction, and each time a pixel transfer clock is received, color data or output setting state data of the corresponding pixel is A shift memory circuit for sequentially storing the data while shifting the memory in the order of transfer time series, and using the color data or output setting state of each memory cell of the shift memory circuit as color data or output setting state data of a corresponding pixel in the mask area. The moving picture processing device according to any one of claims 12 to 15, further comprising a data output unit for outputting. 17. The individual memory cells of the shift memory circuit are arranged in series such that a data output terminal is located on the upstream side in the color data input direction but is connected to a data input terminal on the downstream side. And a D-type flip-flop circuit in which the pixel transfer clock is input to each clock input terminal. A data output line as the data output unit branches from a data output terminal of each D-type flip-flop circuit. Claim 1 provided with
7. The moving image processing device according to 6. 18. A method for setting the mask area so as to include pixels arranged adjacently at right angles to a scanning direction of the moving image frame (hereinafter referred to as pixels arranged in a scanning orthogonal direction).
The memory circuit unit with the transfer control function is arranged so as to correspond to the pixels arranged in the scanning orthogonal direction in a one-to-one correspondence, and stores the color data or the output setting state data of those pixels sequentially and individually in chronological order of transfer. A storage memory unit, and a remaining memory unit (hereinafter, referred to as a subsequent memory unit) except for a data storage memory unit arranged in the array, excluding a memory unit located at the input head (hereinafter, referred to as a head memory unit). And a line memory circuit for storing color data or output setting state data of a pixel on a scanning line immediately before a pixel to be stored in each succeeding memory circuit. The line memory circuit outputs color data or output setting state data of the pixel whose transfer position on the scanning line corresponds to the pixel input to the head memory circuit with respect to the corresponding subsequent memory circuit. Video processing apparatus according to any one of claims 12 to 17 for transferring data.