- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have configured an FPGA design for my linux device that is displaying SDI and HMDI inputs. It was originally configured to receive YCbCr 4:2:2 video format. Since I am experiencing green spill on images when trying to chroma key, I am attempting to change the input data to RGBA.
For a test, I have implemented a conversion into the FPGA that handles the RGB input from the frame buffer. I believe this conversion is correct because the color values come out as expected.
FPGA Design Here:
FB -> CSC -> CRS -> Output
This is the same for all four mixers.
I am using gstreamer to input the color bars pattern into the pipeline.
(gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3")
There are 4 frame buffers located at "/dev/fb0", "/dev/fb1", and so on.
Basically the issue is these frame buffer's graphics are overlapping and changing the memory regions in the DTS file does not fix the issue:
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb0"
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb1"
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3"
I can even see the bottom portion of the color bars pattern when changing the height to 720.
"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=720 ! videoconvert ! filesink location="/dev/fb3"
The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c
Here is the DTS snippet of these frame buffer components and the boot logs where the frame buffers are initialized:
DTS snippet
Boot Logs
The virtual address and length seen in these boot logs do not change at all when DTS changes are made.
I also get this error when inputting a pipeline:
If anyone could help me approach this issue it would be greatly appreciated.
Thanks,
Evan
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Thank you for reaching out.
Apologize for the delayed response as we encounter some technical difficulty.
Just to let you know that Intel has received your support request and currently we are confirming the details with our internal team.
I shall come back to you with findings.
Thank you for your patience.
Best Regards,
ZH_Intel
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
Just to note for internal team and customer:
- DTS and boot are related to HPS
- The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c, this is also driver for HPS
Screenshot
Moving to the right SME for this case.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan
Apologies for the delay, I'm Kian and got assigned to this case. Regarding this issue, I've already contacted our VIP team to get their insight into this issue, will get back to you later once we have findings on our end.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
The team here is asking for more information on what are the base addresses are and the associated settings in the 4 frame buffers? In addition, did you do any maximum bandwidth calculations to check that the EMIF can service all 4 frame buffers at the same time if required?
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Khtan,
The frame buffer's base addresses and associated settings can be seen in the DTS snippet I provided:
In this DTS file image you can see the virtual base addresses, max width and height, bits per symbol, and port width.
Here you can see the frame buffers initializing to their physical addresses:
The EMIF has had no issues before handling all 4 frame buffers, but since I have changed the video format from YCbCr to RGB I guess that could be surpassing the maximum bandwidth. How would I conduct those calculations to check if the EMIF can handle all 4 frame buffers with RGB?
Thanks,
Evan
evanh@javs.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
From the team feedback, referring to the VIP user guide (https://www.intel.com/content/dam/support/us/en/programmable/support-resources/bulk-container/pdfs/literature/ug/archives/ug-vip-18-1.pdf) section 15.7 Motion Adaptive Mode Bandwidth Requirements
Screenshot (just in case you dont have access to the pdf link):
This is the consideration on the bandwidth requirements for video frame data based on deinterlacer, but it is the same process for the frame buffer as well.
Contrast this with the DDR configuration you have configured – width of data bus, DDR speed – use 90% of this figure to account for refresh and inefficiency through the EMIF. If your requirements exceed this 90% value, then this will be the problem. Certainly by changing from YCbCr to RGB this will consume 50% more bandwidth so it sounds highly likely this is the issue.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Kian,
Thank you for your response, I did not see it till now. I am attempting to do these calculations, but I am a little confused on how to calculate for all 4 frame buffers. I'm assuming I would calculate for 1 of them and multiply it by 4, but while doing this I realized I forgot to mention we are sending static images as of right now. The input is a RGB gstreamer pipeline with the default smpte color bars pattern that is in a paused state so the bandwidth shouldn't be exceeded. I could understand this becoming an issue when I try to input a RGB live feed in playing state into the frame buffers, but this should not be the problem while the pipeline is in a paused state. Also, while in paused state the frame rate does not apply, so I am not sure how I would calculate for the paused frame rate.
Thanks
Evan
evanh@javs.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Sorry for the delay in response, I'm trying to confirm some of the information with our expert over here before able to address your question, will get back to you early this week once he is back to office.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Just got the confirmation from the team on the static frames. So basically the video is static is irrelevant to the frame buffer, each pixel still consumes the same amount of bandwidth. As per your reply previously, calculating for one buffer and multiplying by the number of buffers is correct, assuming that all 4 buffers are active at the same time, which is likely in this case since you have all 4 displays up displaying the same thing. Another thing suggested by the team is to try is to increase the amount of available bandwidth by increasing the DDR frequency.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Kian,
I believe you are correct. Our DDR4 is already maxed out at 1200 MHz and from my calculations I believe we are using nearly all of our bandwidth, like around 98%. Do you have any recommendations on what video format I should use? The reason we are navigating away from YCbCr is because our chroma keyer feature cannot handle small detailed graphics. There is always a residual green spill leftover from detailed graphics. The only thing we could think to do to accommodate for this was switching to RGBA, which is why I started conducting those RGB tests with the frame buffers. I will attach a image of the green spill I am talking about. Thank you again for your time and help!
Thank you,
Evan
evanh@javs.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Sorry for the delay in getting back to you as I was out of office till next week. The team suggested that if the DDR is maxed out then you have to use YCbCr but this will suffer from the residual green spill kind of thing. You might be able to mitigate against it by using the color space convertors to go back up to RGBA prior to doing the chroma keying.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Just checking on this issue , whether does the suggestion on using color space convertor able to mitigate the residual green spill?
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Kian,
I believe your suggestion has a good chance of working. I am still working on this new approach. I believe I have the conversion down, but I am not sure if my Chroma Keying logic is good for handling RGB. I am also having to make a custom color plane combiner because I have to separate RGBA and A in order to convert back to YCbCr. The custom component I made combines YCbCr w/ Alpha. I will let you know if I get better results once I get it working.
Thanks,
Evan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Thanks for the the info, I will wait for your results.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey Kian,
I have implemented this new approach into my design. I am inputting YCbCr 4:2:2 through a gstreamer pipeline and converting it to RGB before the chroma keying. I have also changed the chroma keying logic to handle RGB data.
Input YCbCr 4:2:2 -> Convert and resample to RGB 4:4:4 -> Chroma key into RGBA 4:4:4 -> Separate RGB and A -> Convert and resample RGB 4:4:4 to YCbCr 4:2:2 -> Combine YCbCr with Alpha -> Output
I used intel's Color Plane Sequencer to separate RGB and A. I was not able to use this same component to combine YCbCr with Alpha since YCbCr is in 4:2:2 and has different bit sizes for each color space. I made the custom combiner component to handle this.
Y(8 bits) Cb(4 bits) Cr(4 bits)
I have not been getting adequate results so I learnt how to use the Signal Tap Logic Analyzer to debug. The custom combiner I made does not seem to be outputting any data. I will provide my new chroma keyer and custom combiner component. The chroma keyer signals and ports are still the same names as before because I haven't manually changed them yet but the logic has been changed to handle RGB.
The goal is to chroma key RGB 4:4:4 video data to remove the green residuals leftover from chroma keying YCbCr 4:2:2, but the output still needs to receive YCbCr w/ A 4:2:2
Thanks,
Evan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Thanks for the information. Let me check with the team and get back to you.
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Our video team provided some links and commented that
"If the customer has different bit sizes for each color space they will run into trouble. They need to pad or crop color planes so that the IPs see the same number of planes and then use the IPs. If they want to consider adding protocol convertors so they can use the VVP IPs, the bits per color sample does this. "
See :
https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/color-plane-manager.html
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
Just checking up on this case status whether you have any updates?
Thanks
Regards
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evan,
I would like to inquire if there have been any updates.
Looking forward to hearing back from you soon.
Regards,
Kian
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I’m sorry for the late response, we are in the later stages of development for a different project and have been prioritizing it as of late. I haven’t been able to work much more on this yet since my last update. This project is still important to us, and I will let you know as soon as we start back up. I think the VVP IP components could help a lot and save me from making a custom converter from scratch.
Thanks,
Evan

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page