What is the secret to how the VIP Scaler works, and can it handle 4:2:2 video?

[Altera_Forum]

We are currently working on a system which is fairly complex. We thought we could save some time by utilizing Altera's VIP Suite for decimation and interpolation (yuk, yuk). We are using custom filter coefficients and the VIP scalers in polyphase mode for this purpose. We have struggled a bit to get things going due to poor documentation issues and lack of trained support. We are quite not there yet and very frustrated, needless to say. 

 Our biggest concern is if we present 4:2:2, 20-bit video to the system input, how do the scalers know what to filter? How do they know the difference between the chroma and the luma with respect to filter operation? We are currently using the same filters for luma as well as chroma, but we separated them from each other for this reason and possibly the use of different coefficients for each later on. This has opened up a whole other can of worms which I won't go into right now.  All of the reference designs that I have looked at insinuate that the 4:2:2 needs to be upsampled to 4:4:4 before performing any operations on the video. However, we do not want to expend extra chip resources by doing so if we do not have to.  If we need to upsample to get it right, I am OK with it, but I want someone to explain why to me. It seems that the VIP Suite user manual is nothing more than a vague narrative regarding an example design and really does not contain the nuts and bolts type of information that serious video engineers require to make informed decisions.  Please, if anyone out there really knows how the VIP Suite scaler function actually processes data with respect to a 4:2:2 video stream, then I would really like to know.

[Altera_Forum]

the Scaler does not support 4:2:2. you must do chroma resampling to 4:4:4 to scale.

[Altera_Forum]

I've used and am currently using the VIP suite pretty extensively. If you have any questions feel free to PM me. I've not found it so difficult. thepancake is correct that the scaler will not support 4:2:2. Page 5-19 of the user's guide states: 

 Color Pattern: One, two or three channels in sequence or in parallel as selected in the MegaWizard interface. For example, if three channels in sequence is selected where a, b and g can be any color plane: γ β α  So a color pattern in sequence or parallel but not both. So use the Chroma Resampler to upconvert first then scale. You could of course write your own chroma resampler if you'd like. That bit's not too difficult.  Now to answer your question "how the VIP Suite scaler function actually processes data with respect to a 4:2:2 video stream". It doesn't. But if you pass 4:2:2 through, it would not distinguish between Cr and Cb but instead assume them to be one and the same color plane. So your scaled result will have truly interesting values in the Chroma fields.  Jake

[Altera_Forum]

4:2:2 to 4:4:4 upsampling will use hardly any resources. All it is is a registering of the Cr and Cb values so they extended for an extra clock cycle, and the enable on the register toggles for new value. 

 so you end up with this:   Y1 Y2 Y3 Y4 Y5 .... Cb1 Cb2 Cb2 Cb3 Cb3 .... Cr1 Cr1 Cr2 Cr2 Cr3 ....

[Altera_Forum]

To thepancake: Thank you. All I wanted was a direct answer in terms of video engineering technology. You answered the question directly and without a soliloquy. 

 To Tricky: You must not work with video very frequently. What you just described is a nearest neighbor algorithm for upsampling. If you were to examine the output of such a filter on a a diagonal edge it would look horrendous on a hi-def monitor. This would not be acceptable for our purposes. We develop our own filters and coefficients for such work.  To jakobjones: Thank you for your response, but your excerpt from the manual only futher underlines my point regarding the vague nature of how the VIP Suite user guide is written.   I develop video systems and video IP for a living. Altera does not describe how their VIP Suite functions with regard to video processing terminology which is what video engineers utilize. It was obviously written by someone who does not understand video technology otherwise, it would have been stated in the manual, "...does not support 4:2:2 video streams..."   This is the kind of nuts and bolts information that I am talking about that would have saved us time. This has happened to us more than once with regard to use of Altera's other IP. We spend quite a bit of time reading fluff and guessing at the meaning rather than directly discerning it quickly from the text.  I appreciate the offer of using you as a contact. I may take you up on this if required in the future.  As a side note: It would really be an excellent idea for Altera to develop a training module for the Video IP Suite - explaining how it works and translating their terminology into standard video processing terminology for each component as well as the constraints on each component.  Please do not comment on the 30-minute slide show sales pitch for the VIP Suite...it is just as generic and useless as the user guide IMHO.  Thanks Again...

[Altera_Forum]

Well it doesn't sound like the Altera VIP is suitable for your needs. For someone of your experience level I expect that you've already got a scaler written that you've used in other products. Is there a reason why not to use that scaler here? If it's a matter of wrapping the avalon logic around it that's pretty straitforward.  

 I will make mention of something for others. The algorithm mentioned by Tricky is typically referred to as nearest-neighbor without filtering. The Altera Chroma Resampler does support a simple luma-adaptive upconversion which basically applies a simple sinc() filter.  In addition to the additional logic required for the filtering one should always bare in mind that the Altera VIP functions also require logic for the video control packet removal and insertion. So there is additional logic beyond that needed for the video processing.  Jake

[Altera_Forum]

rmatsick, 

 I totally agree with your conclusions on the VIP suite. We had previously done some DVI video processing where we implemented everything in Verilog ourselves, and were able to run through a Cyclone III FPGA at up to 165MHz with only 4-5 pixels worth of latency for an entire complex video processing chain. We recently ran out of funding on a project here due to the fact that we wasted several man months fighting with the VIP cores trying to process NTSC video, only to learn time after time that it did not support 4:2:2 or did not support interlaced video properly. Most of the VIP cores only seem to work well with progressive RGB video. In the long run I am spending my own time off the clock rewriting the project in straight Verilog.  The VIP suite looks great on paper, but there are so many little caveats that are not documented, and my experience is that the timing of the VIP processing chain is very very touchy and requires a lot of carefully implemented constraints to build an FPGA that works properly. And all this was for NTSC video. I would hate to try using it for 165MHz DVI!  I think that the people who are using the VIP successfully are probably using it with one of the larger and faster FPGA's like an Aria or Stratix, and they are using DDR2 or faster frame buffers to help the VIP chain run without underflows.  In my case I have a Cyclone -8 speed grade with SDRAM (16Mx16 at 133MHz). In theory it is plenty of horsepower for processing NTSC video - 27MHz 8 bit or 13.5MHz 16-bit, but sadly I am finding that it is not enough to handle much using the VIP cores.  Jim Morris

[Altera_Forum]

Very much agree with jfmorris. Using VIP sucks. I've wasted plenty of time w/it, sent some service requests to mySupport, and mostly I got the answer finally that the function I try to use does not work with interlaced video, or does work with RGB only, and so on. Implementing the converter IPs eats a lot of logic, so for complex video designs only big chips can be used. 

 The documentation is critically bad, too.

[Altera_Forum]

--- Quote Start ---  Well it doesn't sound like the Altera VIP is suitable for your needs. For someone of your experience level I expect that you've already got a scaler written that you've used in other products. Is there a reason why not to use that scaler here? If it's a matter of wrapping the avalon logic around it that's pretty straitforward.   I will make mention of something for others. The algorithm mentioned by Tricky is typically referred to as nearest-neighbor without filtering. The Altera Chroma Resampler does support a simple luma-adaptive upconversion which basically applies a simple sinc() filter.  In addition to the additional logic required for the filtering one should always bare in mind that the Altera VIP functions also require logic for the video control packet removal and insertion. So there is additional logic beyond that needed for the video processing.  Jake  --- Quote End ---    Jake,  To answer your question. Yes, and yes, and, by the way, yes again! We always end up developing our own custom functions after spending valuable man-weeks of time attempting to use Altera's IP based upon their descriptions of use. (BTW, I am not the only one who is frustrated here...read the other posts...)  The entire reason why we would purchase someone else's IP is to save time! That is the only reason why anyone would purchase HDL IP. You trade time for money. You have obviously missed this point, or tend to develop in a vacuum and without deadlines.  The VIP Suite may be great for university students who are learning how to use Altera's tools and are trying to fulfill their credit requirements, but I question the validity of it's use for a purpose such as ours. Our intention was to use the VIP suite to aid us in fast and reliable feasibility studies. We never intended to use the IP as part of product, only to aid us with idea development.  We also recognize that there are always learning curves for any such new IP. However, when one cannot rely on the documentation or paid subscription support, it becomes an issue regarding the value received with respect to money paid.  I also understand that Altera is in the business of selling chips. The software tools and IP are secondary from a corporate viewpoint. That too, is a no-brainer. However, we are not in this boat either. We perform pure research and use Altera's tools as a vehicle to get to where we want to go, and nothing more. Hence, the lack of dedicated support from Altera.  Regarding the issue of creating wrappers for our own components for insertion into the Avalon fabric: we have done this as well - back with v7.1 when it was easy...however, attempting to reuse any of these components now in v9.1 is impossible.   The avalon interface specifications change with each new version of Quartus. This is quite frustrating and time-consuming as well. What this means to us is a complete re-write of the interface to match the new specification. Specifically, with the advent of the Video-ST interface found in all of the Video IP Suite components, there is no longer direct correlation between legacy Avalon interconnect specifications and the new streaming video interconnect specifications.   Time is money, and changing our components interfaces to match the new specification requires time to research the new spec, time to think about the changes, and time to implement the changes and debug. This is self-defeating. It would have been easier to be able to read documentation, understand documentation, make rational decisions based upon clear documentation, and then decide to implement paid IP or develop custom components.  rick-

[Altera_Forum]

I'm not an Altera employee. However, I have been using the tools for quite some time now (since about 4.1). If there is anything that I or other members of the forum can help you with to make your design successful, don't hesitate to ask.  

 I know there are several companies using the VIP suite in their products (myself included). I'm sure some of them can provide help should you continue to battle with it.  Jake

[Altera_Forum]

--- Quote Start ---  I'm not an Altera employee. However, I have been using the tools for quite some time now (since about 4.1). If there is anything that I or other members of the forum can help you with to make your design successful, don't hesitate to ask.    I know there are several companies using the VIP suite in their products (myself included). I'm sure some of them can provide help should you continue to battle with it.   Jake  --- Quote End ---   I am using vip suite 9.0 and quartus 9.0 for out new project.  I am able to process frames with scaler in static parameters mode there is problem with runtime controle mode.I had following issues  1.I did not found any where a detailed procedure for giving the run time parameters.The vip suite pdf is very vague not detailed in this issue.  2.I could not able to generate the frames using the run time control i could only process single frame that too not correctly  the parameters i am giving are output width 64 and height 32 using the registers  and input width and height as 32x32 using the control packet but the problem is the core is not discarding the extra data if i gave more than 1024 pixels and the output is same size as the input  please help me regarding these issues  Regards ysc

[Altera_Forum]

The runtime control register map for the scaler is given on page A-19 of the user guide. The general description of avalon slave interfaces is given on page 4-14. 

 1 - Which scaling algorithm are you using? 2 - Are you using SoPC builder are interfacing directly to the core? 3 -   --- Quote Start ---  and input width and height as 32x32 using the control packet but the problem is the core is not discarding the extra data if i gave more than 1024 pixels and the output is same size as the input  --- Quote End ---   This concerns me, why do you have "extra data". The length of the video packet should match exactly what's specified in the control packet. Where is the "extra data" coming from? Who is creating the control packets?  The biggest problem I've seen so far that people have had with the VIP suite is not that the cores aren't working but that they misunderstand how the entire video system must be architected.  Jake

[Altera_Forum]

immJake, 

 You are right about that... it is a matter of understanding that you may need more than one piece to get the job done since there are shortcomings with individual pieces and how they are designed and interact with another to perform a specific purpose...  From a video engineer's/architect's perspective the Video IP Suite was not designed properly with respect to the expected function of each piece. However, this could have been easily overcome with good documentation. There are many assumptions that need to made when using the Video IP Suite, and if you are a "newbie" to video processing, I can see the confusion and subsequently frustration immeditately.  Precise documentation (and not power-point presentations or block diagrams) covers a multitude of sins...