US9336565B2 - Image processing device, display apparatus, and image processing method - Google Patents
Image processing device, display apparatus, and image processing method Download PDFInfo
- Publication number
- US9336565B2 US9336565B2 US14/719,914 US201514719914A US9336565B2 US 9336565 B2 US9336565 B2 US 9336565B2 US 201514719914 A US201514719914 A US 201514719914A US 9336565 B2 US9336565 B2 US 9336565B2
- Authority
- US
- United States
- Prior art keywords
- memory
- data
- pixels
- pixel
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 53
- 238000003672 processing method Methods 0.000 title claims description 4
- 238000012937 correction Methods 0.000 claims abstract description 84
- 239000013598 vector Substances 0.000 claims abstract description 56
- 238000000034 method Methods 0.000 claims description 11
- 239000000872 buffer Substances 0.000 description 12
- 230000014509 gene expression Effects 0.000 description 11
- 230000003287 optical effect Effects 0.000 description 8
- 239000000284 extract Substances 0.000 description 7
- 230000006870 function Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 229910052736 halogen Inorganic materials 0.000 description 1
- 150000002367 halogens Chemical class 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- QSHDDOUJBYECFT-UHFFFAOYSA-N mercury Chemical compound [Hg] QSHDDOUJBYECFT-UHFFFAOYSA-N 0.000 description 1
- 229910001507 metal halide Inorganic materials 0.000 description 1
- 150000005309 metal halides Chemical class 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/60—Memory management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/18—Image warping, e.g. rearranging pixels individually
Definitions
- the present invention relates to a technology for correcting images.
- a projector when the positional relationship between a screen and a projection system is deviated from an ideal state, the image projected on the screen is distorted.
- image processing of imparting a distortion opposite to the existing distortion to the input image is used. This image processing is performed for a pixel to be processed of the input image by referring to data of another pixel at a position designated by an offset vector corresponding to the pixel to be processed. In this case, when the components of the offset vector are not integers, the image processing is performed referring to data of four pixels around the position designated by the offset vector.
- JP-A-10-336573 discloses a technique of accessing a memory at a clock rate double the image clock to generate a zoom image using data of two pixels in the vertical direction.
- the above technique has a problem that the necessity of generating a high-speed clock increases the cost of the image processing circuit.
- An advantage of some aspects of the invention is providing a technique of shortening the time of obtaining data used for correction without use of a high-speed clock for memory access.
- an image processing device that processes an image displayed on a display section having pixels of r rows ⁇ c columns.
- the device includes: a first memory section that has a plurality of memory areas equivalent to data of k1 rows of the image and stores data of at least two adjoining pixels in each of the plurality of memory areas; a second memory section that has a plurality of memory areas equivalent to data of k2 rows of the image (k1 and k2 are natural numbers satisfying Omax ⁇ (k1+k2) ⁇ r where Omax is an offset maximum value set previously for the display section) and stores data of at least two adjoining pixels in a row different from pixels of which data is stored in the first memory section in each of the plurality of memory areas; and a correction section that corrects data of an object pixel, out of the pixels of r rows ⁇ c columns, using data of a plurality of pixels stored in a memory area corresponding to a position designated by an offset vector corresponding to the object pixel in the first memory section and the second memory section.
- the time required to obtain data used for correction can be shortened without use of a high-speed clock for memory access.
- the correction section may perform the correction using data stored in the first memory area and the second memory area.
- the correction can be made even when the position designated by the offset vector is located between a pixel stored in the first memory area of the first memory section and a pixel stored in the second memory area of the second memory section.
- the correction section may perform the correction using data sequentially read from the two memory areas.
- the correction can be made even when the position designated by the offset vector is located between two pixels stored in two memory areas of the first memory section.
- the image processing device may further include a cache memory section that stores data used by the correction section in the latest unit period, wherein, when the object pixel has moved by one pixel from the latest unit period, the correction section may perform the correction using data stored in the cache memory section and data stored in either the first memory section or the second memory section.
- the data read time can be more shortened compared with the case that the cache memory section is not used.
- the image processing device may further include: a third memory section that stores data of at least two adjoining pixels including a pixel adjoining, in the same row, a pixel of which data is stored in the first memory section; and a fourth memory section that stores data of at least two adjoining pixels including a pixel adjoining, in the same row, a pixel of which data is stored in the second memory section, wherein the correction section may correct data of the object pixel using data of a plurality of pixels stored in memory areas, in the first memory section, the second memory section, the third memory section, and the fourth memory section, corresponding to a position designated by an offset vector corresponding to the object pixel.
- the data read time can be more shortened compared with the case that the third and fourth memory sections are not used.
- a display apparatus includes: a display section having pixels of r rows ⁇ c columns; a first memory section that has a plurality of memory areas equivalent to data of k1 rows of an image displayed on the display section and stores data of at least two adjoining pixels in each of the plurality of memory areas; a second memory section that has a plurality of memory areas equivalent to data of k2 rows of the image (k1 and k2 are natural numbers satisfying Omax ⁇ (k1+k2) ⁇ r where Omax is an offset maximum value set previously for the display section) and stores data of at least two adjoining pixels in a row different from pixels of which data is stored in the first memory section in each of the plurality of memory areas; and a correction section that corrects data of an object pixel to be processed, out of the pixels of r rows ⁇ c columns, using data of a plurality of pixels stored in a memory area corresponding to a position designated by an offset vector corresponding to the object pixel in the first memory section and the second
- the time required to obtain data used for correction can be shortened without use of a high-speed clock for memory access.
- an image processing method that processes an image displayed on a display section having pixels of r rows ⁇ c columns.
- the method includes: storing data, to a first memory section that has a plurality of memory areas equivalent to data of k1 rows of the image, of at least two adjoining pixels in each of the plurality of memory areas; storing data, to a second memory section that has a plurality of memory areas equivalent to data of k2 rows of the image (k1 and k2 are natural numbers satisfying Omax ⁇ (k1+k2) ⁇ r where Omax is an offset maximum value set previously for the display section), of at least two adjoining pixels in a row different from pixels of which data is stored in the first memory section in each of the plurality of memory areas; and correcting data of an object pixel to be processed, out of the pixels of r rows ⁇ c columns, using data of a plurality of pixels stored in a memory area corresponding to a position designated by an offset vector corresponding to the object pixel in the first memory section and the second
- the time required to obtain data used for correction can be shortened without use of a high-speed clock for memory access.
- FIG. 1 is a block diagram showing a configuration of a display apparatus 1 according to an embodiment.
- FIG. 2 is a view showing a configuration of a controller 12 .
- FIGS. 3A and 3B are views illustrating distortion of an projected image.
- FIGS. 4A and 4B are views illustrating a FIG. 184 that is to be a post-correction target.
- FIG. 5 is a view illustrating allocation of memory areas.
- FIG. 6 is a view showing a concrete example of allocation of memory areas.
- FIG. 7 is a view illustrating a position Pd designated by an offset vector.
- FIG. 8 is a flowchart showing an operation of the display apparatus 1 .
- FIG. 9 is a view showing details of read address generation processing in step S 104 .
- FIG. 10 is a view illustrating a configuration of a cache 1261 .
- FIG. 11 is a view illustrating a pixel used for correction.
- FIG. 12 is a view showing Operation Example 1.
- FIG. 13 is a view showing Operation Example 2.
- FIG. 14 is a view showing Operation Example 3.
- FIG. 15 is a view showing Operation Example 4.
- FIG. 19 is a view illustrating allocation of memory areas to two frame buffers.
- FIG. 20 is a view illustrating correction of the position Pd in Variation 4.
- FIG. 1 is a block diagram showing a configuration of a display apparatus 1 according to an embodiment.
- the display apparatus 1 is a projective display apparatus that projects an image on a windshield of an automobile, a train, or a plane, for example, which is a so-called head-up display (HUD).
- HUD head-up display
- the image projected is an image of a screen of a car navigation system or an image indicating measurement results of various instruments (e.g., a speedometer and a tachometer).
- the display apparatus 1 has a host device 11 , a controller 12 , a projection unit 13 , a memory section 14 , an IF section 15 , and an operation panel 16 .
- the host device 11 controlling the other components of the display apparatus 1 , includes a central processing unit (CPU) 111 , a read only memory (ROM) 112 , and a random access memory (RAM) 113 .
- the CPU 111 is a control device that controls the components of the display apparatus 1 .
- the ROM 112 is a nonvolatile memory device that stores various programs and data.
- the RAM 113 is a volatile memory device that stores data, and functions as a work area when the CPU 111 executes processing.
- the controller 12 is a device that controls the projection unit 13 .
- the controller 12 functions as an image processing device that applies predetermined image processing to an input video signal.
- the controller 12 outputs the image-processed video signal to the projection unit 13 .
- the projection unit 13 projects an image on a screen in accordance with the image-processed video signal.
- the projection unit 13 has a light source, a light modulator 131 , an optical system, and a drive circuit for these elements (illustration is omitted except for the light modulator 131 ).
- the light source may be a lamp such as a high-pressure mercury-vapor lamp, a halogen lamp, and a metal halide lamp, or a solid-state light source such as a light emitting diode (LED) and a laser diode.
- the light modulator 131 is a device that modulates light emitted from the light source in accordance with the video signal, and has a display section such as a liquid crystal panel and a digital mirror device (DMD), for example.
- the light modulator 131 has pixels arranged in a matrix of r rows and c columns, and modulates light for each pixel.
- the optical system is an element that projects the light modulated by the light modulator on the screen, and has a lens and a prism, for example.
- the memory section 14 is a nonvolatile memory device that stores data and a program, such as a hard disk drive (HDD) and a solid-state drive (SSD).
- a program and map data for executing car navigation for example, are stored.
- the host device 11 generates an image for car navigation using the program and data stored in the memory section 14 and outputs the generated image to a subsequent circuit.
- the IF section 15 is an interface that mediates transmission/reception of a signal or data to/from an external device (e.g. various instruments).
- the IF section 15 has a terminal through which the signal or data is transmitted/received to/from an external device.
- the operation panel 16 is an input device via which the user inputs an instruction to the display apparatus 1 , and includes a keyboard, a button, or a touch panel, for example.
- FIG. 2 is a view showing a configuration of the controller 12 .
- the controller 12 has a video RAM (VRAM) 121 , an arbiter 122 , a video input section 123 , a write IF 124 , a read address generation section 125 , a read IF 126 , a correction section 127 , a video output section 128 , and an offset table 129 .
- VRAM video RAM
- the VRAM 121 is a memory (memory section) that has memory areas storing image data of a total of k rows and is used as a frame buffer, in which k is a natural number satisfying O max ⁇ k ⁇ r (1) where Omax is an offset maximum value previously set depending on the light modulator 131 used. The offset maximum value will be described later.
- the VRAM 121 has four frame buffers VRAM 1211 , VRAM 1212 , VRAM 1213 , and VRAM 1214 (examples of the first to fourth memory sections). Note that, when there is no need to distinguish the VRAMs 1211 , 1212 , 1213 , and 1214 from each other, they are collectively referred to as the VRAM 121 .
- Each of the VRAMs 1211 to 1214 is divided into a plurality of unit memory areas. Each unit memory area is specified by an address. Data of at least two pixels adjoining (sequential) in the row direction is stored in each unit memory area. That is, data of at least two sequential pixels adjoining in the row direction is stored in a packed state in a unit memory area specified by a given address. Data read and write are performed with such a plurality of pixels as one unit.
- data of two adjoining pixels is stored in each unit memory area.
- the unit memory area has a memory capacity of 48 bits: data of the pixel smaller in the x coordinate is stored in higher-order 24 bits, and data of the pixel larger in the x coordinate is stored in lower-order 24 bits.
- the relationship between the memory areas of the VRAMs 1211 to 1214 and the pixels will be described later.
- the VRAM 121 does not necessarily have memory areas of k rows ⁇ c columns as a whole.
- the VRAM 121 may have memory areas larger in number than k rows ⁇ c columns. In this case, part of the memory areas of the VRAM 121 may be divided into blocks and used. Otherwise, the memory area of a single RAM as hardware may be divided into four parts and such parts may be used as the VRAMs 1211 to 1214 .
- the VRAM 121 is included in the controller 12 in the example in FIG. 2 , the VRAM 121 may be provided outside the controller 12 .
- the arbiter 122 is a circuit that arbitrates rights to access the VRAM 121 .
- the video input section 123 receives a video signal (input video signal) from the host device 11 , and outputs the input video signal to the write IF 124 .
- the write IF 124 writes the input video signal into the VRAM 121 as data. At this time, the write IF 124 writes the data into a memory area, out of the memory areas of the VRAM 121 , corresponding to a row designated by a write counter.
- the write counter is a parameter that designates the row for which data write is performed.
- the write IF 124 has a register (memory section) (not shown) for storing the write counter.
- the read address generation section 125 generates an address indicating a memory area where data used for correction of data of a pixel to be processed is stored (such an address is hereinafter referred to as a “read address”).
- the read address generation section 125 generates the read address using a read counter and an offset vector.
- the read counter is a parameter designating a row that serves as the reference of data read.
- the read address generation section 125 has a register (memory section) (not shown) for storing the read counter.
- the offset vector is obtained from the offset table 129 , which is a memory storing a table where offset vectors are recorded. The offset vector will be described later.
- the read IF 126 reads data from the VRAM 121 .
- data of two adjoining pixels is included in a memory area designated by one address.
- the read IF 126 outputs data to be used for image processing, out of the read data, to the correction section 124 .
- the read IF 126 has a cache 1261 , which is a memory (an example of the cache memory) that temporarily stores data already read from the VRAM 121 .
- the correction section 127 performs data correction.
- the correction section 127 performs the correction using data stored in a memory area designated by an address generated by the read address generation section 125 , out of the data stored in the VRAM 121 . A concrete method of the correction will be described later.
- the video output section 128 outputs the data corrected by the correction section 127 to the subsequent projection unit 13 as a video signal.
- controller 12 also has an interface that transmits/receives data and instructions to/from the host device 11 .
- FIGS. 3A and 3B are views illustrating distortion of an image projected.
- FIG. 3A illustrates an image displayed on the light modulator 131
- FIG. 3B illustrates an image projected on a screen Sc.
- the light modulator 131 has a display area 181 .
- a rectangle 182 is displayed on the display area 181 .
- a FIG. 183 is projected on the screen Sc.
- the FIG. 183 should be a rectangle if the positional relationship between the optical system of the projection unit 13 and the screen is ideal. In this example, however, since the positional relationship between the optical system and the screen is deviated from the ideal state, the FIG. 183 is distorted from a rectangle.
- the FIG. 183 has points 1831 , 1832 , 1833 , and 1834 corresponding to the vertexes of the rectangle 182 . An outline of a method for solving this distortion will be described hereinafter.
- FIGS. 4A and 4B are views illustrating a FIG. 184 that is to be the post-correction target.
- FIG. 4A shows the FIG. 184 on the screen Sc.
- the FIG. 184 is a figure satisfying, for example, the conditions that (i) it has sides in predetermined directions (e.g., the vertical direction and the horizontal direction), (ii) it has the same aspect ratio as the rectangle 182 , and (iii) a reference position of the FIG. 184 and a reference position of the FIG. 183 are in a predetermined positional relationship (e.g., the center of gravity of the FIG. 184 corresponds with the center of gravity of the FIG. 183 ).
- predetermined directions e.g., the vertical direction and the horizontal direction
- a reference position of the FIG. 184 and a reference position of the FIG. 183 are in a predetermined positional relationship (e.g., the center of gravity of the FIG. 184 corresponds with the center of gravity of the FIG. 183
- FIG. 184 has vertexes 1841 , 1842 , 1843 , and 1844 . It is preferable that the FIG. 184 further satisfy the condition that (iv) it is inscribed in the FIG. 183 from the standpoint of maximizing the area of the post-correction image. In the example of FIG. 4A , however, the condition (iv) is not satisfied to make the FIGS. 183 and 184 easy to discern.
- a vector indicating a pre-correction point to be referred to (e.g., the point 1831 ) with respect to a post-correction point (e.g., the vertex 1841 ) as the reference is referred to as an offset vector. Since the vertexes 1841 , 1842 , 1843 , and 1844 respectively refer to the different points 1831 , 1832 , 1833 , and 1834 , the offset vector is defined at each of the points.
- the offset vectors are uniquely determined if only the positional relationship between the optical system of the projection unit 13 and the screen is determined.
- the offset vectors can be determined by projecting a predetermined test pattern with the projection unit 13 , shooting the projected image with a camera, and analyzing the image.
- the determined offset vectors may be stored in a memory and referred to at the time of correction.
- FIG. 4B shows a FIG. 185 on the light modulator 131 .
- the distortion-corrected FIG. 184 is displayed on the screen Sc.
- the offset vector at a pixel P(x,y) on the display area 181 is herein represented by Vos(x,y).
- Vos(x,y) the coordinates of the pixel at the upper-left corner in the figures are defined as (0,0), and the rightward and downward directions are defined as the positive directions of the x-axis and y-axis, respectively.
- the magnitude of the offset vector is referred to as the offset amount: the magnitudes of the offset vector in the x and y directions are respectively referred to as the x-direction offset amount and the y-direction offset amount.
- the offset amount can assume either of a positive value and a negative value depending on the direction of the offset vector.
- Offset maximum values may be set individually for both the positive and negative directions, or a single offset maximum value may be set in common for both the positive and negative directions.
- Omax in Expression (1) has either of these two values whichever is larger.
- the components of the offset vector Vos are not necessarily integers.
- the position designated by the offset vector Vos is not a pixel itself but a point between a plurality of pixels.
- FIG. 5 is a view illustrating allocation of memory areas, where squares in broken lines represent pixels.
- unit memory areas of the VRAM 121 are allocated every eight pixels of two rows ⁇ four columns. Specifically, the eight pixels are divided into four portions, in two rows ⁇ two columns, each including two pixels adjoining in the row direction (capacity of each unit memory area) as a unit.
- data of the upper-left portion is stored in the VRAM 1211 , data of the upper-right portion in the VRAM 1212 , data of the lower-left portion in the VRAM 1213 , and data of the lower-right portion in the VRAM 1214 .
- VRAM 121 In the VRAM 121 , assuming that a unit memory area in the j-th row and the i-th column is represented by A(i,j), data of a pixel P(4s,2t) and a pixel P(4s+1,2t) is stored in a unit memory area A(s,t) of the VRAM 1211 . Data of a pixel P(4s+2,2t) and a pixel P(4s+3,2t) is stored in a unit memory area A(s,t) of the VRAM 1212 . Data of a pixel P(4s,2t+1) and a pixel P(4s+1,2t+1) is stored in a unit memory area A(s,t) of the VRAM 1213 . Data of a pixel P(4s+2,2t+1) and a pixel P(4s+3,2t+1) is stored in a unit memory area A(s,t) of the VRAM 1214 .
- FIG. 6 is a view showing a concrete example of allocation of memory areas.
- data of a pixel P(0,0) and a pixel P(1,0) is stored in a unit memory area A(0,0)
- data of a pixel P(4,0) and a pixel P(5,0) is stored in a unit memory area A(1,0).
- data of a pixel P(2,0) and a pixel P(3,0) is stored in a unit memory area A(0,0)
- data of a pixel P(6,0) and a pixel P(7,0) is stored in a unit memory area A(1,0).
- VRAM 1213 data of a pixel P(0,1) and a pixel P(1,1) is stored in a unit memory area A(0,0), and data of a pixel P(4,1) and a pixel P(5,1) is stored in a unit memory area A(1,0).
- data of a pixel P(2,1) and a pixel P(3,1) is stored in a unit memory area A(0,0)
- data of a pixel P(6,1) and a pixel P(7,1) is stored in a unit memory area A(1,0).
- FIG. 7 is a view illustrating a position Pd designated by an offset vector.
- the number of times of access to the VRAM 121 can be reduced by using data stored in the cache 1261 , which will be described later in detail.
- the VRAM 121 has memory areas for storing data of 241 rows. The 241 rows are based on the value obtained by doubling the offset maximum value (120 rows in this example) and adding the memory area for a pixel to be processed (1 row in this example) to the doubled value.
- the offset table 129 stores an offset vector for each pixel.
- FIG. 8 is a flowchart showing an operation of the display apparatus 1 .
- step S 101 the write IF 124 writes part of image data into the VRAM 121 .
- the data written at this time is an amount of data equivalent to the offset maximum value, i.e., data of 120 rows in this example. That is, the write IF 124 writes data of the first to 120th rows of the image data into the VRAM 121 .
- the VRAM 121 has memory areas of the first to 241st rows.
- the image data is written in the memory areas of the first to 120th rows.
- step S 102 the read address generation section 125 and the write IF 124 initialize the read counter Cr and the write counter Cw, respectively.
- the read counter Cr and the write counter Cw are counters indicating the positions of rows on the image data, which both assume values of 1 to 480 in this example.
- the difference between the write counter Cw and the read counter Cr is larger than at least the positive-direction offset maximum value.
- step S 103 the read address generation section 125 and the correction section 127 initialize an x counter Cx, which is a parameter for specifying the pixel to be processed.
- the read address generation section 125 and the correction section 127 individually hold their x counters Cx, which are in synchronization with each other with a sync signal.
- the pixel to be corrected is hereinafter represented by a pixel Pc, which has coordinates (Cx,Cr).
- step S 104 the read address generation section 125 generates a read address.
- the position designated by the offset vector Vos with reference to the pixel Pc is represented by the position Pd.
- the coordinates (xd,yd) of the position Pd is represented by Expression (4) below.
- ( xd,yd ) Pc+Vos (4)
- Data of the position Pd(xd,yd) is stored in a memory area of the kd-th row in the VRAM 121 .
- FIG. 9 is a view showing details of the read address generation processing in step S 104 .
- step S 1401 the read address generation section 125 specifies a reference pixel.
- step S 1402 the read address generation section 125 initializes a counter u, which is a counter that specifies an object pixel out of the pixels P 1 to P 4 .
- step S 1403 the read address generation section 125 determines whether data of the pixel P u is stored in the cache 1261 .
- data of a pixel used in the last correction is stored together with the coordinates of the pixel. If it is determined that data of the pixel P u is stored in the cache 1261 (YES in S 1403 ), the read address generation section 125 shifts the processing to step S 1411 . If it is determined that data of the pixel P u is not stored in the cache 1261 (NO in S 1403 ), the read address generation section 125 shifts the processing to step S 1404 .
- step S 1404 the read address generation section 125 generates a read address corresponding to the pixel P u .
- the read address generation section 125 generates the x coordinate x r and the y coordinate y r of the read address using Expressions (6) and (7) below.
- x r quotient(4, x u ) ⁇ 1 (6)
- y r quotient(2, y u ) ⁇ 1 (7)
- quotient (m,n) represents a quotient obtained by dividing n by m.
- step S 1405 the read address generation section 125 specifies a VRAM in which the data of the pixel P u is stored using the following conditions (8) to (11).
- VRAM 1211 or 1213 when mod( x u 4) 0 or 1
- VRAM 1212 or 1214 when mod( x u 4) 2 or 3
- VRAM 1211 or 1212 when mod( y u 2) 0
- VRAM 1213 or 1214 when mod( y u 2) 1 (11)
- step S 1406 the read address generation section 125 determines the number of reference pixels included in the unit memory area designated by the read address.
- the read address generation section 125 determines that data of two reference pixels P u and P u+1 is included in the unit memory area designated by the read address (x r ,y r ).
- the read address generation section 125 shifts the processing to step S 1407 .
- the read address generation section 125 shifts the processing to step S 1409 .
- step S 1407 the read address generation section 125 outputs a data read request to the VRAM 121 .
- This read request is made for the specific VRAM specified in step S 1405 .
- the read request includes the read address and identifiers of the two pixels of which data is stored at this address, out of the pixels P 1 to P 4 .
- step S 1409 the read address generation section 125 outputs a data read request to the VRAM 121 .
- This read request is made for the specific VRAM specified in step S 1405 .
- the read request includes the read address and an identifier of the one pixel of which data is stored at this address, out of the pixels P 1 to P 4 .
- step S 1410 the read address generation section 125 updates the counter u, which is updated as u u+1 in this case.
- step S 1411 the read address generation section 125 outputs a notification that the data of the pixel P u is stored in the cache 1261 to the read IF 126 .
- step S 1412 the read address generation section 125 determines whether the processing has been completed for all the pixels P 1 to P 4 . If it is determined that the processing has been completed for all the pixels P 1 to P 4 (YES in S 1412 ), the read address generation section 125 terminates the processing in step S 104 . If it is determined that there is a pixel, out of the pixels P 1 to P 4 , yet to be processed (NO in S 1412 ), the read address generation section 125 shifts the processing to step S 1403 .
- step S 105 the read IF 126 acquires data from the VRAM 121 .
- the read request output from the read address generation section 125 in step S 1407 or S 1409 is sent to the VRAM 121 via the arbiter 122 .
- the VRAM 121 reads data from a memory area at the designated address and outputs the data to the read IF 126 via the arbiter 122 .
- FIG. 10 is a view illustrating a configuration of the cache 1261 .
- the cache 1261 has memory areas 1262 and 1263 , in each of which data of four pixels of two rows ⁇ two columns is stored. Data used in the last correction is stored in one of the memory areas 1262 and 1263 , and data to be used for the current correction is stored in the other memory area. To simplify the description, assume herein that the last data is stored in the memory area 1262 and the current data is stored in the memory area 1263 .
- the read IF 126 extracts data of the reference pixels from the acquired data. To any data output from the VRAM 121 , an identifier of each reference pixel included in the data is added. When it is indicated that data of the reference pixels P 1 and P 2 is included in the data, the read IF 126 extracts data of the higher-order 24 bits as data of P 1 and data of the lower-order 24 bits as data of P 2 . When it is indicated that data of only the reference pixel P 1 is included in the data, the read IF 126 extracts data of the lower-order 24 bits as data of P. When it is indicated that data of only the reference pixel P 2 is included in the data, the read IF 126 extracts data of the higher-order 24 bits as data of P 2 .
- the read IF 126 extracts data of the higher-order 24 bits as data of P 3 and data of the lower-order 24 bits as data of P 4 .
- the read IF 126 extracts data of the lower-order 24 bits as data of P 3 .
- the read IF 126 extracts data of the higher-order 24 bits as data of P 4 .
- the read IF 126 writes the data of the extracted reference pixels into the memory area 1263 .
- the read IF 126 When being notified by the read address generation section 125 that the data of the pixel P u is stored in the cache 1261 , the read IF 126 reads the data of the pixel P u from the memory area 1262 and writes the read data into the memory area 1263 .
- step S 106 the correction section 127 corrects the data of the pixel Pc using Expression (2).
- FIG. 11 is a view illustrating a pixel used for correction.
- Pc P(14,12)
- Vos(14,12) ( ⁇ 8, ⁇ 10).
- Pd (6,2) (12)
- Dr (14,12) Do (6,2) (13)
- step S 107 the video output section 128 outputs the data of the pixel Pc corrected by the correction section 127 to the subsequent circuit (e.g., the drive circuit of the light modulator 131 ).
- step S 108 the read address generation section 125 and the correction section 127 increment their x counters Cx.
- step S 110 the write IF 124 writes data of the row designated by the write counter Cw into the VRAM 121 .
- the data of the row designated by the write counter Cw is written into a memory area of the kw-th row in the VRAM 121 .
- step S 111 the read address generation section 125 and the write IF 124 increment the counters Cr and Cw.
- a plurality of frames of images may be processed continuously for processing of a moving image.
- the write counter Cw may be initialized, and data of the (i+1)th frame may be written into the VRAM 121 .
- the read counter Cr may be initialized, to start processing of the data of the (i+1)th frame.
- the processing load can be reduced compared with the configuration in which a row including a pixel used for correction is selected and stored in a line buffer. Also, since the read counter and the write counter count at the same speed, such a problem that the read may outpace the write (image data of which write has not been finished may be read mistakenly) will not occur.
- the controller 12 according to this embodiment it is only necessary to provide the controller 12 according to this embodiment between the host device 11 and the projection unit 13 (light modulator 131 ) for a configuration that is not according to the invention (a configuration that excludes the controller 12 from the example of FIG. 1 ).
- the function of correcting distortion can be introduced to a system having no support for distortion correction easily at low cost.
- FIG. 12 is a view showing Operation Example 1.
- the reference pixels P 1 to P 4 are (0,0), (1,0), (0,1), and (1,1), and no data of the reference pixels is stored in the cache 1261 .
- correction is performed using data read from the unit memory area A(0,0) of the VRAM 1211 and the unit memory area A(0,0) of the VRAM 1213 .
- the data to be read is circled in the figure.
- FIG. 13 is a view showing Operation Example 2.
- the reference pixels P 1 to P 4 are (1,0), (2,0), (1,1), and (2,1), and no data of the reference pixels is stored in the cache 1261 .
- correction is performed using data read from the unit memory area A(0,0) of the VRAM 1211 , the unit memory area A(0,0) of the VRAM 1212 , the unit memory area A(0,0) of the VRAM 1213 , and the unit memory area A(0,0) of the VRAM 1214 .
- FIG. 14 is a view showing Operation Example 3.
- the reference pixels P 1 to P 4 are (3,1), (4,1), (3,2), and (4,2), and no data of the reference pixels is stored in the cache 1261 .
- correction is performed using data read from the unit memory area A(1,1) of the VRAM 1211 , the unit memory area A(0,1) of the VRAM 1212 , the unit memory area A(1,0) of the VRAM 1213 , and the unit memory area A(0,0) of the VRAM 1214 .
- FIG. 15 is a view showing Operation Example 4.
- the reference pixels P 1 to P 4 at the last correction are (0,0), (1,0), (0,1), and (1,1)
- the reference pixels P 1 to P 4 at the current correction are (1,0), (2,0), (1,1), and (2,1).
- the cache 1261 data of the pixels (0,0), (1,0), (0,1), and (1,1) is stored.
- data read from the unit memory area A(0,0) of the VRAM 1212 and the unit memory area A(0,0) of the VRAM 1214 and data stored in the cache 1261 are used. In this case, therefore, data used for correction can be obtained with two times of access to the VRAM 121 .
- the offset vector is defined for each pixel, the offset vectors of adjoining pixels are not different largely. In most cases, the reference pixels at the current correction are shifted by one pixel from the reference pixels at the last correction. Therefore, by use of the cache 1261 , the number of times of access to the VRAM 121 can be widely reduced.
- the reference pixels P 1 to P 4 are (0, ⁇ 1), (1, ⁇ 1), (0,0), and (1,0). That is, the reference pixels P 1 and P 2 are pixels falling outside the range of the original image data (hereinafter referred to as “original data”). As data of such a pixel outside the range of the original data, data of a predetermined value is used, which is stored in a register (not shown) in the read IF 126 . When a reference pixel is outside the range of the original data, the read address generation section 125 notifies the read IF 126 of this fact.
- the reference pixels P 1 to P 4 are (0, ⁇ 2), (1, ⁇ 2), (0, ⁇ 1), and (1, ⁇ 1). That is, all the reference pixels are pixels falling outside the range of the original data.
- the read address generation section 125 notifies the correction section 127 of this fact.
- the correction section 127 does not perform correction (interpolation) but outputs a predetermined value as it is as data of the object pixel to be processed.
- VRAMs 1211 to 1214 are described as a single VRAM 121 , not being distinguished from one another.
- the hatched portion indicates that data of the first to 120th rows is stored, and the non-hatched portion indicates that no data is stored (null values are stored).
- both the x component and the y component of the offset vector Vos(Pc) are negative. That is, the reference pixel is a pixel that does not exist in the original data.
- the correction section 127 gives a predetermined value (e.g., a null value or a gray scale corresponding to black) as the post-correction data.
- Vos and Pd are represented by broken lines for the sake of convenience, indicating that a pixel nonexistent in the original data is being referred to.
- data of the first to 241st rows of the image is stored in the memory areas of the first to 241st rows of the VRAM 121 .
- data of the 242nd to 321st rows of the image is stored in the memory areas of the first to 80th rows of the VRAM 121 .
- data of the 81st to 241st rows of the image is stored in the memory areas of the 81st to 241st rows of the VRAM 121 .
- VRAM 121 data of the number of rows equivalent to the offset maximum value is stored in both the positive and negative directions with respect to the forward direction of the read counter, permitting read of data used for correction.
- the number of frame buffers included in the VRAM 121 is not limited to four.
- the VRAM 121 may two frame buffers.
- FIG. 19 is a view illustrating allocation of memory areas in the case of using two frame buffers (VRAM 1211 and VRAM 1212 ).
- Data of a pixel P(2s,2t) and a pixel P(2s+1,2t) is stored in a unit memory area A(s,t) of the VRAM 1211 .
- Data of a pixel P(2s,2t+1) and a pixel P(2s+1,2t+1) is stored in a unit memory area A(s,t) of the VRAM 1212 .
- the VRAM 121 may not include a plurality of frame buffers, but may serve as a single frame buffer.
- the number of reference pixels used for correction is not limited to four, but (m ⁇ n) pixels of m rows and n columns around the position Pd designated by the offset vector may be used as the reference pixels. For example, 16 pixels of four rows ⁇ four columns around the position Pd designated by the offset vector may be used as the reference pixels.
- the memory capacity of the VRAM 121 is not limited to k rows, but the VRAM 121 may have memory areas equivalent to all pixels of one frame, i.e., r rows. In this case, although the capacity of the VRAM 121 is large, the read and write control with the read counter and the write counter is unnecessary.
- FIG. 20 is a view illustrating the correction of the position Pd in Variation 4, where point X indicates the position Pd before the correction, and point Y indicates the position Pd after the correction.
- the read IF 126 may not have the cache 1261 .
- the offset vector Vos can be theoretically defined for all pixels P(x,y). However, when storing offset vectors of all pixels P(x,y) requires a large memory capacity, the offset vectors to be stored may be thinned. For example, offset vectors may be stored in the offset table 129 every two pixels in both the row and column directions. This results in storing offset vectors of only four corner pixels out of nine pixels of three rows ⁇ three columns. For the pixels of which offset vectors are not stored, offset vectors of such pixels may be calculated by interpolation using offset vectors of adjacent pixels of which offset vectors are stored.
- the memory capacity of the VRAM 121 is not limited to that described in the above embodiment.
- the VRAM 121 may have memory areas of the number of rows corresponding to the value obtained by summing the positive-direction offset maximum value, the negative-direction offset maximum value, and the memory area for the pixel to be processed.
- the positive-direction offset maximum value O + max 120
- the negative-direction offset maximum value O ⁇ max 80
- the number of pixels to be processed is 1
- the value of the write counter Cw may be shifted by the negative-direction offset maximum value from the read counter Cr as in Expression (17) below.
- Cw Cr+O ⁇ max (18)
- the display apparatus 1 is not limited to the HUD, but may be a normal projector.
- the invention may be applied to keystone correction of the projector.
- the invention is effective for a small projector such as a so-called pico-projector.
- the configurations of the display apparatus 1 and the components of the display apparatus 1 are not limited to those illustrated in FIGS. 1 and 2 . While an example where hardware (the controller 12 ) executes the flow in FIG. 8 has been described in the above embodiment, the CPU 111 of the host device 11 may execute the flow in FIG. 8 in accordance with a program. In this case, the controller 12 may be omitted. Also, while an example where the read counter and the write counter are stored in registers of the read address generation section 125 and the write IF 124 has been described in the embodiment, these counters may be stored in another memory device such as the VRAM 121 .
- the program to be executed by the CPU 111 may be presented as being stored in a computer-readable recording medium such as a magnetic recording medium (a magnetic tape, a magnetic disk (a HDD and a flexible disk (PD)), etc.), an optical recording medium (an optical disk (a compact disk (CD) and a digital versatile disk (DVD)), etc.), a magneto-optical recording medium, and a semiconductor memory (a flash ROM, etc.).
- a computer-readable recording medium such as a magnetic recording medium (a magnetic tape, a magnetic disk (a HDD and a flexible disk (PD)), etc.), an optical recording medium (an optical disk (a compact disk (CD) and a digital versatile disk (DVD)), etc.), a magneto-optical recording medium, and a semiconductor memory (a flash ROM, etc.).
- a computer-readable recording medium such as a magnetic recording medium (a magnetic tape, a magnetic disk (a HDD and a flexible disk (PD)), etc.), an optical recording medium (an optical disk (a
- the invention may be applied to electronic equipment other than the display apparatus, such as an imaging apparatus.
- the invention may be applied to correction in a digital still camera, a digital video camera, and a drive recorder.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Controls And Circuits For Display Device (AREA)
- Image Processing (AREA)
- Transforming Electric Information Into Light Information (AREA)
- Image Input (AREA)
- Control Of Indicators Other Than Cathode Ray Tubes (AREA)
Abstract
Description
Omax<k<r (1)
where Omax is an offset maximum value previously set depending on the
Dr(P(x,y))=Do(P(x,y)+Vos(x,y)) (2)
x 2 =x 1+1
y 2 =y 1
x 3 =x 1
y 3 =y 1+1
x 4 =x 2
y 4 =y 3
x 1 ≦xd<x 2
y 1 ≦yd<y 2 (3)
(xd,yd)=Pc+Vos (4)
Data of the position Pd(xd,yd) is stored in a memory area of the kd-th row in the
kd−yd(mod k) (5)
For example, in the case that k=241, kd=yd when yd=1 to 241, and kd=2 when yd=243.
x r=quotient(4,x u)−1 (6)
y r=quotient(2,y u)−1 (7)
where quotient (m,n) represents a quotient obtained by dividing n by m.
VRAM 1211 or 1212 when mod(y u2)=0 (10)
Pd=(6,2) (12)
Substituting the above in Expression (2), the following is obtained.
Dr(14,12)=Do(6,2) (13)
kw=Cw(mod k) (14)
For example, in the case that k=241, kw=Cw when Cw=1 to 241, and kw=2 when Cw=243.
Pd=(320,300) (15)
From Expression (5),
kd=59 (16)
Therefore, the data of the reference pixel Pu is stored in the memory area of the 59th row of the
Cw=Cr+O −max (18)
Claims (8)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014112522A JP6361290B2 (en) | 2014-05-30 | 2014-05-30 | Image processing apparatus, display apparatus, and image processing method |
JP2014-112522 | 2014-05-30 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20150348267A1 US20150348267A1 (en) | 2015-12-03 |
US9336565B2 true US9336565B2 (en) | 2016-05-10 |
Family
ID=54702403
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/719,914 Active US9336565B2 (en) | 2014-05-30 | 2015-05-22 | Image processing device, display apparatus, and image processing method |
Country Status (3)
Country | Link |
---|---|
US (1) | US9336565B2 (en) |
JP (1) | JP6361290B2 (en) |
CN (1) | CN105323519A (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5924020B2 (en) * | 2012-02-16 | 2016-05-25 | セイコーエプソン株式会社 | Projector and projector control method |
US11240475B2 (en) | 2018-04-17 | 2022-02-01 | Sony Corporation | Information processing apparatus and method |
CN112130325B (en) * | 2020-09-25 | 2022-07-12 | 东风汽车有限公司 | Parallax correction system and method for vehicle-mounted head-up display, storage medium and electronic device |
JP2023046936A (en) * | 2021-09-24 | 2023-04-05 | シャープディスプレイテクノロジー株式会社 | Display device and manufacturing method of display device |
Citations (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5007099A (en) * | 1987-06-29 | 1991-04-09 | Hitachi, Ltd. | Data filtering method |
JPH03154175A (en) | 1989-11-13 | 1991-07-02 | Toshiba Corp | Picture storage device |
JPH0844862A (en) | 1994-07-29 | 1996-02-16 | Sony Corp | Image processor |
JPH08138039A (en) | 1994-11-10 | 1996-05-31 | Sumitomo Metal Ind Ltd | Semiconductor memory, affine transformation method for picture, picture processor, and address generation circuit |
JPH09147097A (en) | 1995-11-27 | 1997-06-06 | Matsushita Electric Ind Co Ltd | Data interpolating device |
JPH09311934A (en) | 1996-05-24 | 1997-12-02 | Sharp Corp | Image processor |
JPH10336573A (en) | 1997-05-30 | 1998-12-18 | Sanyo Electric Co Ltd | Digital still camera |
JP2000050372A (en) | 1998-07-30 | 2000-02-18 | Hitachi Ltd | Data memory device and data memory control method |
JP2000090257A (en) | 1998-09-11 | 2000-03-31 | Taito Corp | Image processor |
JP2000182067A (en) | 1998-12-16 | 2000-06-30 | Matsushita Electric Ind Co Ltd | Intermediate pixel arithmetic unit |
JP2003271378A (en) | 2002-03-15 | 2003-09-26 | Fuji Xerox Co Ltd | Storage device |
JP2006211402A (en) | 2005-01-28 | 2006-08-10 | Casio Comput Co Ltd | Camera apparatus and image processing method |
JP2012019338A (en) | 2010-07-07 | 2012-01-26 | Renesas Electronics Corp | Semiconductor device |
US8878757B2 (en) * | 2006-09-15 | 2014-11-04 | Semiconductor Energy Laboratory Co., Ltd. | Display device and method of driving the same |
US9035867B2 (en) * | 2007-05-18 | 2015-05-19 | Semiconductor Energy Laboratory Co., Ltd. | Liquid crystal display device and driving method thereof |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3843902B2 (en) * | 2002-07-11 | 2006-11-08 | セイコーエプソン株式会社 | Projector to correct keystone distortion |
JP2009236955A (en) * | 2008-03-25 | 2009-10-15 | Seiko Epson Corp | Projector |
JP5556234B2 (en) * | 2010-02-25 | 2014-07-23 | セイコーエプソン株式会社 | VIDEO PROCESSING CIRCUIT, ITS PROCESSING METHOD, LIQUID CRYSTAL DISPLAY DEVICE, AND ELECTRONIC DEVICE |
JP5744418B2 (en) * | 2010-05-18 | 2015-07-08 | キヤノン株式会社 | Projection apparatus and projection method |
CN102156969B (en) * | 2011-04-12 | 2013-09-25 | 潘林岭 | Processing method for correcting deviation of image |
JP5958079B2 (en) * | 2012-05-22 | 2016-07-27 | 株式会社Jvcケンウッド | Projection apparatus, image correction method, and program |
JP6201290B2 (en) * | 2012-10-02 | 2017-09-27 | セイコーエプソン株式会社 | Image display device and image adjustment method of image display device |
-
2014
- 2014-05-30 JP JP2014112522A patent/JP6361290B2/en not_active Expired - Fee Related
-
2015
- 2015-05-22 US US14/719,914 patent/US9336565B2/en active Active
- 2015-05-29 CN CN201510289890.7A patent/CN105323519A/en active Pending
Patent Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5007099A (en) * | 1987-06-29 | 1991-04-09 | Hitachi, Ltd. | Data filtering method |
JPH03154175A (en) | 1989-11-13 | 1991-07-02 | Toshiba Corp | Picture storage device |
JPH0844862A (en) | 1994-07-29 | 1996-02-16 | Sony Corp | Image processor |
JPH08138039A (en) | 1994-11-10 | 1996-05-31 | Sumitomo Metal Ind Ltd | Semiconductor memory, affine transformation method for picture, picture processor, and address generation circuit |
JPH09147097A (en) | 1995-11-27 | 1997-06-06 | Matsushita Electric Ind Co Ltd | Data interpolating device |
JPH09311934A (en) | 1996-05-24 | 1997-12-02 | Sharp Corp | Image processor |
US6762792B1 (en) | 1997-05-30 | 2004-07-13 | Sanyo Electric Co., Ltd. | Digital still camera |
JPH10336573A (en) | 1997-05-30 | 1998-12-18 | Sanyo Electric Co Ltd | Digital still camera |
JP2000050372A (en) | 1998-07-30 | 2000-02-18 | Hitachi Ltd | Data memory device and data memory control method |
JP2000090257A (en) | 1998-09-11 | 2000-03-31 | Taito Corp | Image processor |
JP2000182067A (en) | 1998-12-16 | 2000-06-30 | Matsushita Electric Ind Co Ltd | Intermediate pixel arithmetic unit |
JP2003271378A (en) | 2002-03-15 | 2003-09-26 | Fuji Xerox Co Ltd | Storage device |
JP2006211402A (en) | 2005-01-28 | 2006-08-10 | Casio Comput Co Ltd | Camera apparatus and image processing method |
US8878757B2 (en) * | 2006-09-15 | 2014-11-04 | Semiconductor Energy Laboratory Co., Ltd. | Display device and method of driving the same |
US9035867B2 (en) * | 2007-05-18 | 2015-05-19 | Semiconductor Energy Laboratory Co., Ltd. | Liquid crystal display device and driving method thereof |
JP2012019338A (en) | 2010-07-07 | 2012-01-26 | Renesas Electronics Corp | Semiconductor device |
Also Published As
Publication number | Publication date |
---|---|
JP2015228056A (en) | 2015-12-17 |
JP6361290B2 (en) | 2018-07-25 |
US20150348267A1 (en) | 2015-12-03 |
CN105323519A (en) | 2016-02-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8322862B2 (en) | Projector, computer program product, and trapezoidal distortion correcting method | |
JP5348022B2 (en) | Projector and projection conversion processing apparatus | |
US9336565B2 (en) | Image processing device, display apparatus, and image processing method | |
JP5997882B2 (en) | Projector and projector control method | |
JP6155717B2 (en) | Image processing apparatus, projector, and image processing method | |
CN114125411B (en) | Projection device correction method, projection device correction device, storage medium and projection device | |
US9761160B2 (en) | Image processing device, display apparatus, image processing method, and program | |
US10657622B2 (en) | Controlling projected image frame rate in response to determined projection surface curvature | |
JP6418010B2 (en) | Image processing apparatus, image processing method, and display apparatus | |
JP5348035B2 (en) | projector | |
CN109634032B (en) | Image processing method and device | |
US20220292652A1 (en) | Image generation method and information processing device | |
JP5533109B2 (en) | Image display device, projector, and data acquisition method in image display device | |
JP2019106589A (en) | Projector, geometric distortion correction method, and program | |
JP2018088669A (en) | Projection image distortion correction device and method | |
JP7418690B2 (en) | Display control device, display system, display control method and program | |
JP2012230302A (en) | Image generation device, projection type image display device, image display system, image generation method and computer program | |
JP2010288062A (en) | Projector, program, information storage medium, and image projection method | |
JP5353772B2 (en) | projector | |
JP6064699B2 (en) | Image processing apparatus, projector, and image processing method | |
US11924587B2 (en) | Display method, projector, and projection system | |
US20210165205A1 (en) | Image processing apparatus, image processing program, and image processing method | |
JP5413265B2 (en) | Image display device and projector | |
JP4266150B2 (en) | Projection apparatus and projection method | |
US20230209026A1 (en) | Projection method and projector |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SEIKO EPSON CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HAYASHI, TADASHI;REEL/FRAME:035699/0837 Effective date: 20150331 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |