CN107623804A - A kind of method of terminal device and photographing panorama picture - Google Patents
A kind of method of terminal device and photographing panorama picture Download PDFInfo
- Publication number
- CN107623804A CN107623804A CN201610556503.6A CN201610556503A CN107623804A CN 107623804 A CN107623804 A CN 107623804A CN 201610556503 A CN201610556503 A CN 201610556503A CN 107623804 A CN107623804 A CN 107623804A
- Authority
- CN
- China
- Prior art keywords
- camera module
- processing unit
- picture
- terminal device
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 107
- 241000251468 Actinopterygii Species 0.000 claims abstract description 78
- 230000000007 visual effect Effects 0.000 claims abstract description 34
- 238000012937 correction Methods 0.000 claims description 23
- 238000005520 cutting process Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 8
- 238000004148 unit process Methods 0.000 claims description 6
- 230000002596 correlated effect Effects 0.000 claims description 3
- 210000003128 head Anatomy 0.000 description 15
- 230000006870 function Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000003384 imaging method Methods 0.000 description 6
- 239000011521 glass Substances 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000000875 corresponding effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Abstract
The invention discloses a kind of method of terminal device and photographing panorama picture, wherein terminal device have two arrange opposite to each other possess fish-eye camera module (or possess the camera lens module of two non-flakes positioned opposite but outside be socketed with fish eye lens), respectively with two camera modules, two graphics processing units being connected and the processor being connected with two graphics processing units.Two camera modules can open simultaneously, so, when the first camera module and second camera module are opened simultaneously, the image that the first camera module arranged opposite to each other and second camera module gather is spliced, can collection reaches the image i.e. panoramic picture in 360 ° of angulars field of view, the photo and video for shooting full visual angle are realized, improves the usage experience of user.
Description
Technical field
The present invention relates to camera technical field, the method for more particularly to a kind of terminal device and photographing panorama picture.
Background technology
Existing intelligent terminal all possesses camera such as mobile phone, tablet personal computer, and user can carry out whenever and wherever possible
Take pictures and share network, or the functions such as Internet video chat are carried out using camera, be greatly enriched making for user
With experience.
Common intelligent terminal can be including two cameras：Front camera and rear camera, front camera and
Rear camera structure is similar, one towards user, one, backwards to user, includes in front camera and rear camera
Camera lens module, one group of eyeglass is further included in camera lens module, however, being front camera or rear camera, it shoots
Visual angle it is all limited, typically between 40 degree to 80 degree, also, front camera and rear camera synchronization are merely able to make
With one of them, therefore, current camera can not photograph the photo or video at full visual angle.
The content of the invention
In view of the above problems, it is proposed that the present invention so as to provide one kind overcome above mentioned problem or at least in part solve on
The method for stating a kind of terminal device and photographing panorama picture of problem.
In a first aspect, the embodiments of the invention provide a kind of terminal device, including：First camera module and the second shooting
Head mould group, the first graphics processing unit being connected with the first camera module, the second graph being connected with second camera module
Processing unit, and the processor being connected with first graphics processing unit and second graph processing unit；First camera
Module and second camera module are arranged and can opened simultaneously opposite to each other；
Include a fish eye lens in first camera module and second camera module, and the fish eye lens is the
The camera lens of outermost end in one camera module and second camera module；
First graphics processing unit and second graph processing unit, it is respectively used to image the first camera module and second
The picture signal of head mould group collection is handled；
The processor, for when the first camera module and second camera module are opened simultaneously, to the first figure
Image after processing unit and second graph processing unit processes is spliced, and spliced image is exported.
A camera lens is included in first camera module and second camera module, the camera lens is fish eye lens.
In one embodiment, at least two camera lenses, institute are included in the first camera module and second camera module
The camera lens stated at least two camera lenses positioned at outermost end is fish eye lens；
Camera lens on the inside of the fish eye lens is non-fish eye lens.
In one embodiment, first graphics processing unit, it is additionally operable to when the first camera module and the second shooting
When the first camera module is only opened in head mould group, the picture signal collected to the first camera module carries out distortion correction；
The second graph processing unit, it is additionally operable to when only opening the in the first camera module and second camera module
During two camera modules, the picture signal collected to second camera module carries out distortion correction；
The processor, it is additionally operable to only open one of them in the first camera module and second camera module
When, the picture signal after the first graphics processing unit or second graph processing unit processes is exported.
In one embodiment, first graphics processing unit or second graph processing unit, specifically for for
Each frame of video in picture or video that one camera module and second camera module collect, it is determined that wherein needing
The distorted region removed；The picture or frame of video are circle；The distorted region determined is removed；It is described abnormal to removing
The picture or frame of video become outside region is stretched, and it is rectangle to make the picture after stretching or frame of video.
In one embodiment, first graphics processing unit or second graph processing unit, specifically under
The mode of stating determines to need the distorted region removed：It is determined that the inscribed square in the circular picture or frame of video；According to
The default circular arc number of degrees, determine in the inscribed square between each two adjacent vertex to recessed four in the center of circle direction
Circular arc；The distortion that the region surrounded between four circular arcs and the picture or the circumference of frame of video is removed as needs
Region.
In one embodiment, the size of distorted region for needing to remove and the first camera module for being started or
Fish-eye focal length size is negatively correlated in person's second camera module.
In one embodiment, first graphics processing unit or second graph processing unit, specifically for removing
The picture or frame of video of the distorted region are stretched, and it is that the length of side is equal to the circular arc to make the picture after stretching or frame of video
Rectangle.
In one embodiment, it is also associated with the first figure between first camera module and the first graphics processing unit
As signal processing unit ISP；The second image letter is also associated between the second camera module and second graph processing unit
Number processing unit ISP.
In one embodiment, the processor, it is the digital signal processor DSP and/or central processing of terminal device
Device CPU.
Second aspect, the embodiment of the present invention additionally provide a kind of terminal device, including：
Terminal device body and the cutting ferrule being removably socketed on the terminal device body；The terminal device sheet
Body includes：The first camera module and second camera module being disposed opposite to each other, be connected with first camera module
One graphics processing unit, the second graph processing unit being connected with the second camera module；And with first figure
The processor that processing unit connects with second graph processing unit；First camera module and second camera module use non-fish
Glasses head and it can open simultaneously；
At least two fish eye lenses being disposed opposite to each other are provided with the cutting ferrule, and when the cutting ferrule is socketed on the terminal
When on apparatus body, at least two fish eye lens is connected in first camera module and second camera module respectively
On；
First graphics processing unit and second graph processing unit, be respectively used to first camera module and
The picture signal that second camera module is gathered by fish eye lens is handled；
The processor, for when the first camera module and second camera module are opened simultaneously, to the first figure
Image after processing unit and second graph processing unit processes is spliced, and spliced image is exported.
The third aspect, the embodiments of the invention provide a kind of terminal device photographing panorama picture provided using first aspect
Method, including：
Instructed according to the shooting at the full visual angle of input, while start the first camera module and the second shooting of terminal device
Head mould group is shot；
First camera module is obtained respectively and picture signal that second camera module collects；
The picture signal collected to first camera module and second camera module is spliced, and generates panorama
Exported after image.
In one embodiment, methods described also includes：
Instructed according to the shooting at the non-full visual angle of input, start one in the first camera module and second camera module
It is individual；
Obtain the first camera module and picture or video that second camera module collects；
The picture or video collected to the first camera module and second camera module carries out distortion correction；
By by the picture or video frequency output of distortion correction.
Fourth aspect, the embodiments of the invention provide a kind of terminal device photographing panorama picture provided using second aspect
Method, including：
Instructed according to the shooting at the full visual angle of input, while start the first camera module and the second shooting of terminal device
Head mould group is shot；
First camera module is obtained respectively and second camera module is collected by the fish eye lens on cutting ferrule
Picture signal；
The picture signal collected to first camera module and second camera module by the fish eye lens
Spliced, exported after generating panoramic picture.
Technical scheme provided in an embodiment of the present invention can include the following benefits：
In the method for above-mentioned terminal device and photographing panorama picture provided in an embodiment of the present invention, terminal device has two
The fish-eye camera module that possesses arranged opposite to each other (or possesses the camera lens module of two non-flakes positioned opposite still
Outside is socketed with fish eye lens), two graphics processing units being connected respectively with two camera modules and with two figures
The processor of processing unit connection.Two camera modules can open simultaneously, the first camera module and second camera mould
The fish eye lens included in group is circumscribed with flake for the camera lens of outermost end or the first camera module and second camera module
Camera lens camera lens, so, because fish eye lens is a kind of extreme wide-angle lens, the shooting visual field is bigger, so, when the first shooting
Head mould group and second camera module are opened simultaneously when, the first camera module arranged opposite to each other and second camera module are adopted
The image of collection is spliced, it is possible to which collection reaches the image in 360 ° of angulars field of view, realizes the photo for shooting full visual angle and regards
Frequently, the usage experience of user is improved.
Further, the method for terminal device and photographing panorama picture provided in an embodiment of the present invention, it is also an option that property
Ground only opens one in the first camera module and second camera module, and then, the first camera module and second are taken the photograph
The picture or video shot as head mould group by fish eye lens carries out distortion correction, and the picture after distortion correction is exported,
So that user also disclosure satisfy that the demand of user's normal photographing when need not carry out full viewing angles.
Other features and advantages of the present invention will be illustrated in the following description, also, partly becomes from specification
Obtain it is clear that or being understood by implementing the present invention.The purpose of the present invention and other advantages can be by the explanations write
Specifically noted structure is realized and obtained in book, claims and accompanying drawing.
Below by drawings and examples, technical scheme is described in further detail.
Brief description of the drawings
Accompanying drawing is used for providing a further understanding of the present invention, and a part for constitution instruction, the reality with the present invention
Apply example to be used to explain the present invention together, be not construed as limiting the invention.In the accompanying drawings：
Fig. 1 is the structural representation for the terminal device 1 that the embodiment of the present invention one provides；
Fig. 2 is the arrangement signal of the first camera module 11 that the embodiment of the present invention one provides and second camera module 12
Figure；
Fig. 3 is the first camera module 11 that the embodiment of the present invention one provides and the another arrangement of second camera module 12
Schematic diagram；
Fig. 4 is the flow chart for the distorted region that the determination needs that the embodiment of the present invention one provides remove；
The signal of distorted region removed is needed in the picture by flake shooting that Fig. 5 provides for the embodiment of the present invention one
Figure；
Fig. 6 is the another structural representation for the terminal device that the embodiment of the present invention one provides；
Fig. 7 is the flow chart of the method for the terminal device photographing panorama picture that the embodiment of the present invention one provides；
Fig. 8 be the embodiment of the present invention one provide when the first camera module and second camera module only open one
Process chart；
Fig. 9,10 be the structural representation for the terminal device that the embodiment of the present invention two provides；
Figure 11 is the flow chart of the method for the terminal device photographing panorama picture that the embodiment of the present invention two provides；
Figure 12 is network architecture schematic diagram of the terminal device of the offer of the embodiment of the present invention two for VR when live.
Embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although the disclosure is shown in accompanying drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
Limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
Completely it is communicated to those skilled in the art.
Terminal device provided in an embodiment of the present invention is illustrated first.Terminal device bag provided in an embodiment of the present invention
Include two kinds：A kind of is itself just to possess two camera modules being disposed opposite to each other, and the owned fish of the two camera modules
Glasses head, so, open the image that can gathers ultra-wide angle respectively simultaneously by the two camera modules, both are spliced
It can be obtained by panoramic picture.Another is owned two camera modules being disposed opposite to each other, the two shooting head moulds
Group itself uses non-fish eye lens, but is circumscribed with cutting ferrule, when two camera modules are opened simultaneously, passes through external flake
Camera lens, the image of ultra-wide angle can be collected, both are spliced and can be obtained by panoramic picture.Separately below to both situations
Terminal device illustrate.
Embodiment one：
Terminal device 1 provided in an embodiment of the present invention, as shown in figure 1, including：First camera module 11 and the second shooting
Head mould group 12, the first graphics processing unit 13 being connected with the first camera module 11, it is connected with second camera module 12
Second graph processing unit 14, and the processing being connected with first graphics processing unit 13 and second graph processing unit 14
Device 15；First camera module 11 and second camera module 12 are arranged and can opened simultaneously opposite to each other；
Include a fish eye lens, and the fish eye lens in first camera module 11 and second camera module 12
For the camera lens of outermost end in the first camera module 11 and second camera module 12；
First graphics processing unit 13 and second graph processing unit 14, it is respectively used to the first camera module 11 and
The picture signal of two camera modules 12 collection is handled；
Processor 15, for when the first camera module 11 and second camera module 12 are opened simultaneously, to the first figure
Image after shape processing unit 13 and second graph processing unit 14 are handled is spliced, and spliced image is exported.
Terminal device 1 provided in an embodiment of the present invention, can be the various equipment for possessing shoot function, including it is but unlimited
In：Mobile phone, tablet personal computer, all kinds of video cameras or wearable device etc., the various equipment examples for possessing shoot function.This
Inventive embodiments are not limited this.
It should be noted that in above-mentioned terminal device 1 provided in an embodiment of the present invention, the first camera module 11 and second
Arranged opposite to each other between camera module 12, such as one is used as front camera, another as rear camera, or
It is arranged on the opposed face of any two of terminal device.
First camera module 11 and second camera module 12 its structure can have a variety of, and concrete structure may be referred to existing
There is the structure of various camera modules in technology.Mainly lens assembly is included in first camera module and second camera module
With imaging sensor (such as CMOS or CCD).For example, for example may be used in the first camera module and second camera module
With comprising：Lens assembly, imaging sensor, optical filter, flexible printed circuit board (PCB) etc., wherein, imaging sensor fixes peace
Loaded on below lens assembly, for optical filter between lens assembly and imaging sensor, imaging sensor is located at the one of flexible PCB
Side and it is connected with flexible PCB.Analog-to-digital conversion (A/D) converter is also included in camera module, for imaging sensor to be gathered
Analog signal be converted to corresponding data signal.
First camera module 11 and the structure of second camera module 12 can be with identical or differ.
Wherein the first camera module 11 and second camera module 12 can be back-to-back arrangements (as shown in Figure 2), i.e.,
The central shaft of first camera module 11 and second camera module 12 is on the same line or close proximity；Can also be as schemed
Shown in 3, it can also be that the first camera module 11 and second camera module 12 are opposite but are staggeredly arranged, concrete condition can be with
According to the inner space determined by layout of terminal device, the embodiment of the present invention is not limited this.
Above-mentioned terminal device provided in an embodiment of the present invention, is improved prior art, the first camera module 11
Arrange and can open simultaneously opposite to each other with second camera module 12, in the first camera module 11 and second camera module 12
Comprising fish eye lens be outermost end camera lens, so, because fish eye lens is a kind of bigger camera lens in visual field, so, when
First camera module 11 and second camera module 12 are opened simultaneously when, the first camera module 11 and second for arranging opposite to each other
Camera module 12 can shoot the picture or video of ultra-wide angle by respective internal fish eye lens, can after splicing
Reach the picture or video in 360 ° of angulars field of view with collection, realize the photo and video for shooting full visual angle.
Further, the first camera module 11 or the structure of the lens assembly in second camera module 12 can have
Two kinds, a kind of situation is that a camera lens is included in the lens assembly of the first camera module 11 or second camera module 12,
The camera lens is fish eye lens.
Another situation is, in the lens assembly of the first camera module 11 or second camera module 12, comprising
At least two camera lenses, such as a fish eye lens, one or more non-fish eye lens (such as standard lens, wide-angle can be included
Camera lens etc.), but among this at least two camera lens, the camera lens positioned at outermost end is fish eye lens, correspondingly, fish eye lens
The camera lens of inner side is non-fish eye lens.
Non- fish eye lens can use standard lens of the prior art or wide-angle lens first-class, be usually contained in these camera lenses
Multiple eyeglasses (glass or glass lens), typically in 45 °~150 ° of scope, (specific visual angle size has with lens focus at visual angle
Close) in.In itself limited viewing angle of non-fish eye lens, the shooting at full visual angle can not be realized, but when non-fish eye lens has been superimposed flake
After camera lens, its visual angle shot greatly increases, therefore, when the first camera module and second camera module arranged opposite to each other
When opening simultaneously, the camera module can just shoot the picture or video at full visual angle.
First camera module 11 and the structure of the lens assembly in second camera module 12 can with it is identical can not also
Together, the image of ultra-wide angle but can be gathered.
Above-mentioned first graphics processing unit and second graph processing unit, in the specific implementation, prior art can be used
In graph image unit (Graphics Processing Unit, GPU) chip realize that GPU chips are responsible for image operation
The microprocessor of work.
Processor 15 shoots obtained picture and video to the first camera module 11 and second camera module 12 respectively
After being spliced, the picture or video at full visual angle can be obtained, the splicing of picture and video can be carried out in encoded,
Splicing re-encoding can first be completed.
In embodiments of the present invention, fish eye lens is a kind of special bugeye lens, and " fish eye lens " is being commonly called as it,
Protruded for the Camera angle for making camera lens reach maximum, the front lens diameter of this phtographic lens and in parabolical to camera lens is anterior,
It is rather similar to the eyes of fish, " fish eye lens " therefore and gain the name.The maximum effect of fish eye lens is that angular field of view is big, and visual angle can
Reach or close to 180~235 degree, this is to shoot a wide range of Landscape Creation condition.
Above-mentioned terminal device provided in an embodiment of the present invention, it can also only open the first camera according to the needs of user
Module or only second camera module, for example, the first camera module and second camera module one that are disposed opposite to each other are
Front camera, one when being rear camera, user can select to open wherein by default select button or button
Any one.In this case, it is necessary to which the picture signal gathered to the first camera module or second camera module is carried out
Distortion correction processing.
Specifically, above-mentioned first graphics processing unit 13, it is additionally operable to when the first camera module 11 and second camera
When the first camera module 11 is only opened in module 12, the picture signal collected to the first camera module 11 enters line distortion school
Just；
Second graph processing unit 14, it is additionally operable to when only unlatching in the first camera module 11 and second camera module 12
During second camera module 12, the picture signal collected to second camera module 12 carries out distortion correction；
Processor 15, it is additionally operable to only open one of them in the first camera module 11 and second camera module 12
When, the picture signal output after the first graphics processing unit 13 or second graph processing unit 14 are handled.
In one embodiment, the first graphics processing unit 13 or second graph processing unit 14, specifically for for
Each frame of video in picture or video that one camera module and second camera module collect, it is determined that wherein needing
The distorted region removed；The picture or frame of video are circle；The distorted region determined is removed；It is described abnormal to removing
The picture or frame of video become outside region is stretched, and it is rectangle to make the picture after stretching or frame of video.
Further, the first graphics processing unit 13 or second graph processing unit 14, specifically in the following manner
It is determined that need the distorted region removed：It is determined that the inscribed square in the circular picture or frame of video；According to default
The circular arc number of degrees, determine in the inscribed square between each two adjacent vertex to four recessed circular arcs of the center of circle direction；
The distorted region that the region surrounded between four circular arcs and the picture or the circumference of frame of video is removed as needs.
Further, the size for the distorted region that above-mentioned needs remove is taken the photograph with the first camera module started and second
As fish-eye focal length size is negatively correlated in head mould group.
Further, above-mentioned first graphics processing unit 13 or second graph processing unit 14, specifically for abnormal to removing
The picture or frame of video for becoming region are stretched, and it is rectangle of the length of side equal to circular arc to make the picture after stretching or frame of video.
As a result of the image (picture or video) of the first camera module 11 or the shooting of second camera module 12
Due to having reached super large visual angle, barrel distortion is clearly.In order to show non-fish-eye display effect to user, it is also necessary to
Distortion correction is carried out to the picture by fish eye lens shooting or video, the picture that will be distorted or video carry out " evening up ".
Fish-eye correction has a variety of methods, for example with panorama picture of fisheye lens model, completes image plane to thing sphere
Coordinate transform, with etc. arc length mapping method the image of sphere is projected on cylinder, so as to complete fish eye lens fish eye lens
The distortion correction of the picture of shooting.
In another example following methods：Using fish eye lens panorama picture of fisheye lens in the eyes in spherical coordinate model, establish standard
Coordinate system, the conventional coordinates assume that the fish-eye video camera of fish eye lens is placed on the origin of coordinates, and shooting direction is OZ axles
Positive direction, the fish eye images after shooting fall in OXY planes, by (target regards flat on the pixel-map of sphere to target view plane
Face be in OZ axle positive directions, it is parallel with OXY planes and be fish eye lens object lens spherical radius with the distance between OXY planes
Plane).Then the distance between each pixel and the origin of coordinates on OXY is established to regard with the target with respective pixel on target view plane
The mapping of distance between the origin of frequency face, then using corrected parameter, is modified to the pixel in OXY planes, finally gives mesh
The picture on view plane is marked, the picture on this target view plane eventually passes through picture or video after distortion correction.
Above-mentioned each method amount of calculation is bigger, and the embodiment of the present invention additionally provides a kind of more easy bearing calibration, should
The characteristics of position that method occurs substantially to distort using fish eye lens is positioned in picture (circle) outer peripheral areas, will first be distorted
Region remove, then the picture or frame of video that eliminate the distorted region are stretched, make the picture after stretching or
Frame of video is rectangle.
Because user is under the demand of non-full viewing angles, it is however generally that, it is only necessary in the picture of fish eye lens shooting
Be located relatively at central area part content can meet demand, along with occurring substantially to distort positioned at being normally at picture
Outer peripheral areas, therefore, after the region of distortion is first removed, to remove distorted region picture or frame of video stretch, count
Calculation amount is relatively small, can improve the efficiency of the picture distortion correction of fish eye lens shooting.
During stretching, according to the quantity of the pixel at the edge of picture before stretching, it may be determined that the video finally stretched
The size of frame.
For a simply example, it is determined that need the distorted region that removes to obtain step, as shown in figure 4, can be according to
Following step is realized：
Step S41, the inscribed square in circular picture or the frame of video by fish eye lens shooting is determined；
Step S42, according to the default circular arc number of degrees, it is determined that to center of circle side between each two adjacent vertex in inscribed square
To four recessed circular arcs；
Step S43, the region surrounded between four circular arcs and picture or the circumference of frame of video is removed as needs
Distorted region.
As shown in figure 5, the shadow region that the edge of four circular arcs and the picture of fish eye lens shooting surrounds is gone for needs
The distorted region removed.
The angle of circular arc can select according to the size of used fish-eye focal length, and overall selection principle can
With the image-forming principle with reference to lens, i.e. the focal length of camera lens is longer, and its visual angle is smaller, and in requisition for the big of the distorted region of removal
Small also smaller, the radian of circular arc is smaller, that is, needs the size and fish-eye focal length size negative correlation of distorted region removed.
Such as visual angle be 180 ° fish eye lens shoot picture carry out distorted region removal when, the arc of its four circular arcs
Degree, when the picture that the fish eye lens that visual angle is 235 ° is shot that compares carries out distorted region removal, the radian of four circular arcs is small.
As shown in figure 5, being stretched to removing the picture behind the distorted region or frame of video, make the picture or video of stretching
Frame is the rectangle (rectangle on right side in Fig. 5) that the length of side is equal to the circular arc.
It is preferred that the various image processing algorithms of prior art can be used in elongation process, such as inserted using various
Value-based algorithm so that the picture after stretching is smoother, avoids distortion.
For terminal device provided in an embodiment of the present invention, the first camera module 11 and the first graphics processing unit
The first image signal processing unit ISP is also associated between 13；The second camera module 12 and second graph processing unit
The second image signal processing unit ISP is also associated between 14.
Image signal processing unit (Image Signal Processing, ISP) main function is that front-end image is sensed
The unit that the signal of device output is handled, such as noise reduction, signal format conversion etc., to be adapted to the image sensing of different vendor
Device.Specifically refer to prior art.
Above-mentioned processor, in the specific implementation, central processor CPU of the prior art or data signal can be used
(DSP, Digital Signal Processor) is handled to realize, or is realized in the embodiment of the present invention and located in the lump using both
Manage the function of device.
Such as in the terminal device shown in Fig. 6, the first camera module and second camera module respectively with two ISP cores
Piece connects, and each ISP chips are all connected with a GPU chip, and two GPU chips are connected with dsp chip, and dsp chip is set with terminal
Standby CPU connections.
In the specific implementation, above-mentioned ISP chips, GPU chips, dsp chip and CPU can be integrated in same chipset and work as
In.It can also be each independent chip.
For above-mentioned terminal device provided in an embodiment of the present invention, user can want any of shooting at it
Side, realizes the shooting of the picture or video at full visual angle, the picture at the full visual angle of shooting or video can also be shared into other use
Family, greatly improve usage experience of the user to terminal device.
Based on identical inventive concept, the embodiment of the present invention additionally provides a kind of above-mentioned terminal device photographing panorama picture
Before method, it is similar to aforementioned terminals equipment to solve the principle of problem by this method, therefore the implementation of this method may refer to
The implementation of terminal device is stated, part is repeated and repeats no more.
The embodiment of the present invention additionally provides a kind of method of above-mentioned terminal device photographing panorama picture, as shown in fig. 7, the party
Method includes：
Step S71, according to the shooting at the full visual angle of input instruct, while start terminal device the first camera module and
Second camera module is shot；
Step S72, first camera module is obtained respectively and picture signal that second camera module collects；
Step S73, the picture signal collected to first camera module and second camera module is spliced,
Exported after generation panoramic picture.
Based on the structure of above-mentioned terminal device, in above-mentioned steps S71, the shooting that full visual angle is have issued as user instructs,
Such as default button or button are pressed by touching, send and ask the shooting at full visual angle to instruct, terminal device refers to according to the shooting
Order, while start the first camera module and second camera module, due to the first camera module and second camera module
It is disposed opposite to each other and both outermost camera lenses is fish eye lens, so, while starts the first camera module and the second shooting
After head mould group, the image of the super wide viewing angle of the first camera module and the shooting of second camera module can be obtained.
In above-mentioned steps S73, obtained picture is shot to the first camera module and second camera module respectively and is regarded
After frequency is spliced, the picture or video at full visual angle can be obtained, the splicing of picture and video can be carried out in encoded,
Splicing re-encoding can also first be completed.
Further, when user need not carry out full viewing angles, above-mentioned shooting image provided in an embodiment of the present invention
Method, as shown in figure 8, can also carry out following step：
Step S81, instructed according to the shooting at the non-full visual angle of input, start the first camera module and second camera mould
One in group；
Step S82, the picture or video that the first camera module of acquisition and second camera module collect；
Step S83, the picture or video collected to the first camera module and second camera module enters line distortion school
Just；
Step S84, by by the picture or video frequency output of distortion correction.
In step S82, the fish-eye picture or video of shooting are due to having reached super large visual angle, and barrel distortion is very
Substantially.In order to show non-fish-eye display effect to user, it is necessary to the picture or video by fish eye lens shooting
Distortion correction is carried out, the picture of distortion or video are subjected to " evening up ".
The method of distortion correction may be referred to the embodiment of above-mentioned terminal device, will not be repeated here.
Embodiment two：
The terminal device that the embodiment of the present invention two provides, such as Fig. 9, shown in Figure 10, including：
Terminal device body 1 ' and the cutting ferrule 2 ' being removably socketed on the terminal device body 1 '；Wherein：
Terminal device body 1 ' includes：The first camera module 11 ' and second camera module 12 ' being disposed opposite to each other, with
First graphics processing unit 13 ' of the first camera module 11 ' connection, is connected with the second camera module 12 '
Second graph processing unit 14 '；And the processing being connected with the first graphics processing unit 13 ' and second graph processing unit 14 '
Device 15 '；First camera module 11 ' and second camera module 12 ' can be opened using non-fish eye lens and simultaneously；
Cutting ferrule 2 ' is above provided with least two fish eye lenses 21 ' being disposed opposite to each other and fish eye lens 22 ', and when cutting ferrule 2 ' covers
Be connected to terminal device body 1 ' it is upper when, at least two fish eye lenses 21 ' and 22 ' be connected in respectively the first camera module 11 ' with
On second camera module 12 '；
First graphics processing unit 13 ' and second graph processing unit 14 ', are respectively used to the first camera module 11 '
The picture signal gathered with second camera module 12 ' by fish eye lens 21 ' and 22 ' is handled；
Processor 15 ', for when the first camera module 11 ' and second camera module 12 ' while when opening, to first
Image after graphics processing unit 13 ' and second graph processing unit 14 ' are handled is spliced, and spliced image is defeated
Go out.
Above-mentioned terminal device, it is different from the terminal device structure that embodiment one provides, but can realize and be disposed opposite to each other
Camera open simultaneously, while gather the image of ultra-wide angle, and splicing synthesis is carried out to the image of shooting, generate panorama sketch
Picture.
The process of specific collection image and stitching image is referred to the explanation of embodiment one, will not be repeated here.
The terminal device that the embodiment of the present invention two provides, can also be similar with embodiment one, only opens one of them
Camera, plane picture is then obtained by way of distortion correction.Concrete mode can be found in embodiment one.
The embodiment of the present invention two additionally provides the side of the terminal device photographing panorama picture provided using above-described embodiment two
Method, as shown in figure 11, this method include：
S110, instructed according to the shooting at the full visual angle of input, while start the first camera module and the of terminal device
Two camera modules are shot；
S111, the first camera module is obtained respectively and second camera module is collected by the fish eye lens on cutting ferrule
Picture signal；
S112, the picture signal collected to the first camera module and second camera module by fish eye lens are carried out
Splicing, exported after generating panoramic picture.
The implementation of the above method, the embodiment of device is referred to, repeats part and repeat no more.
Whether the terminal device that the terminal device or embodiment two that above-described embodiment one provides provide, all possesses shooting
The function of panoramic picture, so, the image (example that user can be carried with using terminal or external fish eye lens pans
Such as virtual reality (VR) photo or VR videos) and it is sent to network side, you can VR images or VR are realized at low cost, whenever and wherever possible
Video it is live.The network architecture as shown in figure 12, terminal device can by the VR videos of shooting by network upload server,
Other client end of playing back can watch VR net casts by the server.
In the method for above-mentioned terminal device and photographing panorama picture provided in an embodiment of the present invention, the embodiment of the present invention provides
Above-mentioned terminal device and photographing panorama picture method in, terminal device have two arrange opposite to each other possess it is fish-eye
Camera module (or possess the camera lens module of two non-flakes positioned opposite but outside be socketed with fish eye lens), respectively
With two camera modules, two graphics processing units being connected and the processor being connected with two graphics processing units.Two
Camera module can open simultaneously, the fish eye lens or first included in the first camera module and second camera module
Camera module and the external fish eye lens of second camera module are a kind of extreme bugeye lenses, so, are taken the photograph when first
When being opened as head mould group and second camera module simultaneously, by the first camera module arranged opposite to each other and second camera module
The image gathered by fish eye lens is spliced, it is possible to which collection reaches the image in 360 ° of angulars field of view, realizes that shooting is complete
The photo and video at visual angle, improve the usage experience of user.
Further, the method for terminal device and photographing panorama picture provided in an embodiment of the present invention, it is also an option that property
Ground only opens one in the first camera module and second camera module, and then, the first camera module and second are taken the photograph
The picture or video shot as head mould group by fish eye lens carries out distortion correction, and the picture after distortion correction is exported,
So that user also disclosure satisfy that the demand of user's normal photographing when need not carry out full viewing angles.
It should be understood by those skilled in the art that, embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the present invention can use the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the present invention can use the computer for wherein including computer usable program code in one or more
The shape for the computer program product that usable storage medium is implemented on (including but is not limited to magnetic disk storage and optical memory etc.)
Formula.
The present invention is the flow with reference to method according to embodiments of the present invention, equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram
Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
The processors of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which produces, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, so as in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
Obviously, those skilled in the art can carry out the essence of various changes and modification without departing from the present invention to the present invention
God and scope.So, if these modifications and variations of the present invention belong to the scope of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to comprising including these changes and modification.
Claims (14)
- A kind of 1. terminal device, it is characterised in that including：First camera module and second camera module and the first shooting First graphics processing unit of head mould group connection, the second graph processing unit that is connected with second camera module, and with institute State the processor that the first graphics processing unit connects with second graph processing unit；First camera module and second camera mould Group is opposite to arrange and can open simultaneously；Include a fish eye lens in first camera module and second camera module, and the fish eye lens is taken the photograph for first As the camera lens of outermost end in head mould group and second camera module；First graphics processing unit and second graph processing unit, are respectively used to the first camera module and second camera mould The picture signal of group collection is handled；The processor, for when the first camera module and second camera module are opened simultaneously, to the first graphics process Image after unit and second graph processing unit processes is spliced, and spliced image is exported.
- 2. terminal device as claimed in claim 1, it is characterised in that in the first camera module and second camera module Comprising a camera lens, the camera lens is fish eye lens.
- 3. terminal device as claimed in claim 1, it is characterised in that in the first camera module and second camera module Comprising at least two camera lenses, the camera lens at least two camera lens positioned at outermost end is fish eye lens；Camera lens on the inside of the fish eye lens is non-fish eye lens.
- 4. the terminal device as described in claim any one of 1-3, it is characterised in that first graphics processing unit, also use In when only opening the first camera module in the first camera module and second camera module, the first camera module is adopted The picture signal collected carries out distortion correction；The second graph processing unit, it is additionally operable to when only unlatching second is taken the photograph in the first camera module and second camera module During as head mould group, the picture signal collected to second camera module carries out distortion correction；The processor, it is additionally operable to, by when only opening one of them in the first camera module and second camera module, incite somebody to action Picture signal output after first graphics processing unit or second graph processing unit processes.
- 5. terminal device as claimed in claim 4, it is characterised in that first graphics processing unit or second graph processing Unit, regarded specifically for each in the picture or video that are collected for the first camera module and second camera module Frequency frame, it is determined that wherein needing the distorted region removed；The picture or frame of video are circle；By the distortion area determined Domain removes；Stretched to removing picture outside the distorted region or frame of video, make the picture after stretching or the frame of video be Rectangle.
- 6. terminal device as claimed in claim 5, it is characterised in that first graphics processing unit or second graph processing Unit, specifically for determining to need the distorted region removed in the following manner：It is determined that the circular picture or frame of video In inscribed square；According to the default circular arc number of degrees, determine in the inscribed square between each two adjacent vertex to institute State four recessed circular arcs of center of circle direction；The area that will be surrounded between four circular arcs and the picture or the circumference of frame of video The distorted region that domain removes as needs.
- 7. terminal device as claimed in claim 6, it is characterised in that the size of the distorted region for needing to remove is with being opened Fish-eye focal length size is negatively correlated in the first dynamic camera module or second camera module.
- 8. the terminal device as described in claim 5 or 6, it is characterised in that first graphics processing unit or second graph Processing unit, specifically for being stretched to the picture or frame of video that remove the distorted region, make the picture after stretching or regard Frequency frame is the rectangle that the length of side is equal to the circular arc.
- 9. terminal device as claimed in claim 1, it is characterised in that first camera module and the first graphics process list The first image signal processing unit ISP is also associated between member；The second camera module and second graph processing unit it Between be also associated with the second image signal processing unit ISP.
- 10. terminal device as claimed in claim 1, it is characterised in that the processor, be at the data signal of terminal device Manage device DSP and/or central processor CPU.
- A kind of 11. terminal device, it is characterised in that including：Terminal device body and the cutting ferrule being removably socketed on the terminal device body；The terminal device body bag Include：The first camera module and second camera module being disposed opposite to each other, the first figure being connected with first camera module Shape processing unit, the second graph processing unit being connected with the second camera module；And with first graphics process The processor that unit connects with second graph processing unit；First camera module and second camera module use non-flake mirror Head and can open simultaneously；At least two fish eye lenses being disposed opposite to each other are provided with the cutting ferrule, and when the cutting ferrule is socketed on the terminal device When on body, at least two fish eye lens be connected in respectively first camera module and second camera module it On；First graphics processing unit and second graph processing unit, it is respectively used to first camera module and second The picture signal that camera module is gathered by fish eye lens is handled；The processor, for when the first camera module and second camera module are opened simultaneously, to the first graphics process Image after unit and second graph processing unit processes is spliced, and spliced image is exported.
- 12. a kind of method of terminal device photographing panorama picture as described in claim any one of 1-11, it is characterised in that bag Include：Instructed according to the shooting at the full visual angle of input, while start the first camera module and second camera mould of terminal device Group is shot；First camera module is obtained respectively and picture signal that second camera module collects；The picture signal collected to first camera module and second camera module is spliced, and generates panoramic picture After export.
- 13. method as claimed in claim 12, it is characterised in that methods described also includes：Instructed according to the shooting at the non-full visual angle of input, start one in the first camera module and second camera module；Obtain the first camera module and picture or video that second camera module collects；The picture or video collected to the first camera module and second camera module carries out distortion correction；By by the picture or video frequency output of distortion correction.
- A kind of 14. method of terminal device photographing panorama picture as claimed in claim 13, it is characterised in that including：Instructed according to the shooting at the full visual angle of input, while start the first camera module and second camera mould of terminal device Group is shot；First camera module is obtained respectively and figure that second camera module is collected by the fish eye lens on cutting ferrule As signal；The picture signal collected to first camera module and second camera module by the fish eye lens is carried out Splicing, exported after generating panoramic picture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610556503.6A CN107623804A (en) | 2016-07-14 | 2016-07-14 | A kind of method of terminal device and photographing panorama picture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610556503.6A CN107623804A (en) | 2016-07-14 | 2016-07-14 | A kind of method of terminal device and photographing panorama picture |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107623804A true CN107623804A (en) | 2018-01-23 |
Family
ID=61087318
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610556503.6A Pending CN107623804A (en) | 2016-07-14 | 2016-07-14 | A kind of method of terminal device and photographing panorama picture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107623804A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108259765A (en) * | 2018-03-27 | 2018-07-06 | 努比亚技术有限公司 | A kind of image pickup method, terminal and computer readable storage medium |
| CN108495043A (en) * | 2018-04-28 | 2018-09-04 | Oppo广东移动通信有限公司 | Image processing method and relevant apparatus |
| CN109600547A (en) * | 2018-11-30 | 2019-04-09 | Oppo广东移动通信有限公司 | Photographic method, device, electronic equipment and storage medium |
| CN110197569A (en) * | 2018-11-19 | 2019-09-03 | 广东小天才科技有限公司 | A kind of safety monitoring method and wearable device based on wearable device |
| TWI678920B (en) * | 2018-05-23 | 2019-12-01 | 宏碁股份有限公司 | Video processing apparatus, video processing method thereof and computer program product |
| CN112788226A (en) * | 2021-03-16 | 2021-05-11 | 深圳传音控股股份有限公司 | Mobile device, photographing method, and readable storage medium |
| CN116320765A (en) * | 2023-04-06 | 2023-06-23 | 如你所视(北京)科技有限公司 | Method, apparatus, device, medium and program product for generating panoramic image |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020036649A1 (en) * | 2000-09-28 | 2002-03-28 | Ju-Wan Kim | Apparatus and method for furnishing augmented-reality graphic using panoramic image with supporting multiuser |
| CN101072288A (en) * | 2007-06-15 | 2007-11-14 | 林进灯 | Method for obtaining fish-eye image correction relationship and fish-eye correction |
| KR20100127027A (en) * | 2009-05-25 | 2010-12-03 | 연세대학교 산학협력단 | Hdri-panorama image generation apparatus and method with fisheye lens |
| JP2011182176A (en) * | 2010-03-01 | 2011-09-15 | Toyama Univ | Wide viewing angle image processing method, and wide viewing angle image photographing device |
| KR20140048764A (en) * | 2012-10-16 | 2014-04-24 | 재단법인 씨윗코리아 | Panorama camera |
| CN105049569A (en) * | 2015-08-20 | 2015-11-11 | 擎亚国际贸易（上海）有限公司 | Panoramic photo composition method based on front and back cameras of cellphone |
| CN105554478A (en) * | 2016-02-24 | 2016-05-04 | 深圳市车米云图科技有限公司 | Video monitoring system and method |
| CN205847395U (en) * | 2016-07-14 | 2016-12-28 | 幸福在线（北京）网络技术有限公司 | A kind of terminal unit |
-
2016
- 2016-07-14 CN CN201610556503.6A patent/CN107623804A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020036649A1 (en) * | 2000-09-28 | 2002-03-28 | Ju-Wan Kim | Apparatus and method for furnishing augmented-reality graphic using panoramic image with supporting multiuser |
| CN101072288A (en) * | 2007-06-15 | 2007-11-14 | 林进灯 | Method for obtaining fish-eye image correction relationship and fish-eye correction |
| KR20100127027A (en) * | 2009-05-25 | 2010-12-03 | 연세대학교 산학협력단 | Hdri-panorama image generation apparatus and method with fisheye lens |
| JP2011182176A (en) * | 2010-03-01 | 2011-09-15 | Toyama Univ | Wide viewing angle image processing method, and wide viewing angle image photographing device |
| KR20140048764A (en) * | 2012-10-16 | 2014-04-24 | 재단법인 씨윗코리아 | Panorama camera |
| CN105049569A (en) * | 2015-08-20 | 2015-11-11 | 擎亚国际贸易（上海）有限公司 | Panoramic photo composition method based on front and back cameras of cellphone |
| CN105554478A (en) * | 2016-02-24 | 2016-05-04 | 深圳市车米云图科技有限公司 | Video monitoring system and method |
| CN205847395U (en) * | 2016-07-14 | 2016-12-28 | 幸福在线（北京）网络技术有限公司 | A kind of terminal unit |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108259765A (en) * | 2018-03-27 | 2018-07-06 | 努比亚技术有限公司 | A kind of image pickup method, terminal and computer readable storage medium |
| CN108259765B (en) * | 2018-03-27 | 2020-08-11 | 努比亚技术有限公司 | Shooting method, terminal and computer readable storage medium |
| CN108495043A (en) * | 2018-04-28 | 2018-09-04 | Oppo广东移动通信有限公司 | Image processing method and relevant apparatus |
| CN108495043B (en) * | 2018-04-28 | 2020-08-07 | Oppo广东移动通信有限公司 | Image data processing method and related device |
| TWI678920B (en) * | 2018-05-23 | 2019-12-01 | 宏碁股份有限公司 | Video processing apparatus, video processing method thereof and computer program product |
| US10803905B2 (en) | 2018-05-23 | 2020-10-13 | Acer Incorporated | Video processing apparatus, video processing method thereof and non-transitory computer readable medium |
| CN110197569A (en) * | 2018-11-19 | 2019-09-03 | 广东小天才科技有限公司 | A kind of safety monitoring method and wearable device based on wearable device |
| CN109600547A (en) * | 2018-11-30 | 2019-04-09 | Oppo广东移动通信有限公司 | Photographic method, device, electronic equipment and storage medium |
| CN112788226A (en) * | 2021-03-16 | 2021-05-11 | 深圳传音控股股份有限公司 | Mobile device, photographing method, and readable storage medium |
| CN116320765A (en) * | 2023-04-06 | 2023-06-23 | 如你所视(北京)科技有限公司 | Method, apparatus, device, medium and program product for generating panoramic image |
| CN116320765B (en) * | 2023-04-06 | 2024-01-26 | 如你所视(北京)科技有限公司 | Method, apparatus, device, medium and program product for generating panoramic image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107623804A (en) | A kind of method of terminal device and photographing panorama picture | |
| CN109064397B (en) | Image stitching method and system based on camera earphone | |
| CN107666606B (en) | Binocular panoramic picture acquisition methods and device | |
| CN109828345B (en) | Optical image capturing lens, image capturing device and electronic device | |
| TWI424188B (en) | Wide-angle imaging lens assembly | |
| CN103916582B (en) | A kind of image processing method and device | |
| TWI451358B (en) | Banana codec | |
| US8023009B2 (en) | Imaging apparatus for correcting optical distortion and wide-angle distortion | |
| CN205847395U (en) | A kind of terminal unit | |
| WO2016101742A1 (en) | Light field collection control methods and apparatuses, light field collection devices | |
| CN107835372A (en) | Imaging method, device, mobile terminal and storage medium based on dual camera | |
| JP6802372B2 (en) | Shooting method and terminal for terminal | |
| WO2018192579A1 (en) | Camera module | |
| CN106210547A (en) | A kind of method of pan-shot, Apparatus and system | |
| WO2022262344A1 (en) | Photographing method and electronic device | |
| TW201351959A (en) | Method of stereo 3D image synthesis and related camera | |
| CN108107649A (en) | A kind of periscopic camera module and intelligent terminal for intelligent terminal | |
| CN111405087A (en) | Electronic product and camera shooting method thereof | |
| Kagawa et al. | A three‐dimensional multifunctional compound‐eye endoscopic system with extended depth of field | |
| CN107967665A (en) | Image processing method and image processing apparatus | |
| TWM373507U (en) | Three-dimensional vision panoramic image splicing mosaics device | |
| CN115546043B (en) | Video processing method and related equipment thereof | |
| CN108307090A (en) | A kind of camera module, mobile terminal and its method and apparatus for shooting image | |
| JP2012182738A (en) | Stereo image pickup apparatus | |
| CN117278839A (en) | Shooting method, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |