FPGA Intellectual Property
PCI Express*, Networking and Connectivity, Memory Interfaces, DSP IP, and Video IP
6545 Discussions

Problem W/ Altera VIP Frame Buffer II

EvanHamiltonJAVS
3,810 Views

I have configured an FPGA design for my linux device that is displaying SDI and HMDI inputs. It was originally configured to receive YCbCr 4:2:2 video format. Since I am experiencing green spill on images when trying to chroma key, I am attempting to change the input data to RGBA. 

 

For a test, I have implemented a conversion into the FPGA that handles the RGB input from the frame buffer. I believe this conversion is correct because the color values come out as expected.

 

FPGA Design Here:

FPGA_Forum_Help.png

fpgahelppt2.png

FB -> CSC -> CRS -> Output

This is the same for all four mixers.

 

I am using gstreamer to input the color bars pattern into the pipeline.

(gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3")

There are 4 frame buffers located at "/dev/fb0", "/dev/fb1", and so on.

 

Basically the issue is these frame buffer's graphics are overlapping and changing the memory regions in the DTS file does not fix the issue:

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb0"

EvanHamiltonJAVS_0-1715010416346.jpeg

 

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb1"

EvanHamiltonJAVS_1-1715010451602.jpeg

 

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=1080 ! videoconvert ! filesink location="/dev/fb3"

EvanHamiltonJAVS_2-1715010488853.jpeg

 

I can even see the bottom portion of the color bars pattern when changing the height to 720.

"gst-launch-1.0 videotestsrc ! videoconvert ! video/x-raw,format=BGR,width=1920,height=720 ! videoconvert ! filesink location="/dev/fb3"

EvanHamiltonJAVS_3-1715010571472.jpeg

 

The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c

Here is the DTS snippet of these frame buffer components and the boot logs where the frame buffers are initialized:

DTS snippetDTS snippetBoot LogsBoot Logs

The virtual address and length seen in these boot logs do not change at all when DTS changes are made.

 

I also get this error when inputting a pipeline:

nospaceerror.png

 

If anyone could help me approach this issue it would be greatly appreciated.

 

Thanks,

Evan

0 Kudos
22 Replies
ZH_Intel
Employee
3,218 Views

Hi Evan,

 

Thank you for reaching out.

Apologize for the delayed response as we encounter some technical difficulty.

Just to let you know that Intel has received your support request and currently we are confirming the details with our internal team.

I shall come back to you with findings.

 

Thank you for your patience.

 

Best Regards,

ZH_Intel



ZH_Intel
Employee
3,084 Views

Hi All,

 

Just to note for internal team and customer:

  • DTS and boot are related to HPS
  • The drivers being used for the frame buffer is altvipfb.c and altvipfb2-plat.c, this is also driver for HPS

ScreenshotScreenshot

 

Moving to the right SME for this case.

 

 

0 Kudos
khtan
Employee
2,993 Views

Hi Evan

Apologies for the delay, I'm Kian and got assigned to this case. Regarding this issue, I've already contacted our VIP team to get their insight into this issue, will get back to you later once we have findings on our end.

 

Thanks

Regards

Kian

0 Kudos
khtan
Employee
2,970 Views

Hi Evan,

The team here is asking for more information on what are the base addresses are and the associated settings in the 4 frame buffers? In addition, did you do any maximum bandwidth calculations to check that the EMIF can service all 4 frame buffers at the same time if required?

 

Thanks

Regards

Kian

0 Kudos
EvanHamiltonJAVS
2,915 Views

Hey Khtan,

 

The frame buffer's base addresses and associated settings can be seen in the DTS snippet I provided:

 

EvanHamiltonJAVS_0-1718631580632.png

In this DTS file image you can see the virtual base addresses, max width and height, bits per symbol, and port width.

 

Here you can see the frame buffers initializing to their physical addresses:

EvanHamiltonJAVS_1-1718631676923.png

 

 

The EMIF has had no issues before handling all 4 frame buffers, but since I have changed the video format from YCbCr to RGB I guess that could be surpassing the maximum bandwidth. How would I conduct those calculations to check if the EMIF can handle all 4 frame buffers with RGB?

 

Thanks,

Evan

 

evanh@javs.com

0 Kudos
khtan
Employee
2,876 Views

Hi Evan,

From the team feedback,  referring to the VIP user guide (https://www.intel.com/content/dam/support/us/en/programmable/support-resources/bulk-container/pdfs/literature/ug/archives/ug-vip-18-1.pdf) section 15.7 Motion Adaptive Mode Bandwidth Requirements

 

Screenshot (just in case you dont have access to the pdf link):

khtan_0-1718853121631.png

khtan_1-1718853160514.png

This is the consideration on the bandwidth requirements for video frame data based on deinterlacer, but it is the same process for the frame buffer as well.

Contrast this with the DDR configuration you have configured – width of data bus, DDR speed – use 90% of this figure to account for refresh and inefficiency through the EMIF.  If your requirements exceed this 90% value, then this will be the problem. Certainly by changing from YCbCr to RGB this will consume 50% more bandwidth so it sounds highly likely this is the issue.

 

Thanks

Regards

Kian

EvanHamiltonJAVS
2,811 Views

Hi Kian,

 

Thank you for your response, I did not see it till now. I am attempting to do these calculations, but I am a little confused on how to calculate for all 4 frame buffers. I'm assuming I would calculate for 1 of them and multiply it by 4, but while doing this I realized I forgot to mention we are sending static images as of right now. The input is a RGB gstreamer pipeline with the default smpte color bars pattern that is in a paused state so the bandwidth shouldn't be exceeded. I could understand this becoming an issue when I try to input a RGB live feed in playing state into the frame buffers, but this should not be the problem while the pipeline is in a paused state. Also, while in paused state the frame rate does not apply, so I am not sure how I would calculate for the paused frame rate.

 

Thanks

Evan

 

evanh@javs.com

0 Kudos
khtan
Employee
2,728 Views

Hi Evan,

Sorry for the delay in response, I'm trying to confirm some of the information with our expert over here before able to address your question, will get back to you early this week once he is back to office.

 

Thanks

Regards

Kian

0 Kudos
khtan
Employee
2,698 Views

Hi Evan,

Just got the confirmation from the team on the static frames. So basically the video is static is irrelevant to the frame buffer, each pixel still consumes the same amount of bandwidth.  As per your reply previously, calculating for one buffer and multiplying by the number of buffers is correct, assuming that all 4 buffers are active at the same time, which is likely in this case since you have all 4 displays up displaying the same thing.  Another thing suggested by the team is to try is to increase the amount of available bandwidth by increasing the DDR frequency.

 

Thanks

Regards

Kian

 

 

EvanHamiltonJAVS
2,679 Views

Hi Kian,

 

I believe you are correct. Our DDR4 is already maxed out at 1200 MHz and from my calculations I believe we are using nearly all of our bandwidth, like around 98%. Do you have any recommendations on what video format I should use? The reason we are navigating away from YCbCr is because our chroma keyer feature cannot handle small detailed graphics. There is always a residual green spill leftover from detailed graphics. The only thing we could think to do to accommodate for this was switching to RGBA, which is why I started conducting those RGB tests with the frame buffers.  I will attach a image of the green spill I am talking about. Thank you again for your time and help!


Thank you,

Evan

evanh@javs.com

0 Kudos
khtan
Employee
2,579 Views

Hi Evan,

Sorry for the delay in getting back to you as I was out of office till next week. The team suggested that if the DDR is maxed out then you have to use YCbCr but this will suffer from the residual green spill kind of thing. You might be able to mitigate against it by using the color space convertors to go back up to RGBA prior to doing the chroma keying.

 

Thanks

Regards

Kian

0 Kudos
khtan
Employee
2,473 Views

Hi Evan,

Just checking on this issue , whether does the suggestion on using color space convertor able to mitigate the residual green spill?

 

Thanks

Regards

Kian

0 Kudos
EvanHamiltonJAVS
2,455 Views

Hey Kian,

 

I believe your suggestion has a good chance of working. I am still working on this new approach. I believe I have the conversion down, but I am not sure if my Chroma Keying logic is good for handling RGB.  I am also having to make a custom color plane combiner because I have to separate RGBA and A in order to convert back to YCbCr. The custom component I made combines YCbCr w/ Alpha. I will let you know if I get better results once I get it working. 

 

Thanks,

Evan

0 Kudos
khtan
Employee
2,437 Views

Hi Evan,

Thanks for the the info, I will wait for your results.

 

Thanks

Regards

Kian

0 Kudos
EvanHamiltonJAVS
2,353 Views

Hey Kian,

 

I have implemented this new approach into my design. I am inputting YCbCr 4:2:2 through a gstreamer pipeline and converting it to RGB before the chroma keying. I have also changed the chroma keying logic to handle RGB data.

 

Input YCbCr 4:2:2 -> Convert and resample to RGB 4:4:4 -> Chroma key into RGBA 4:4:4 -> Separate RGB and A -> Convert and resample RGB 4:4:4 to YCbCr 4:2:2 -> Combine YCbCr with Alpha -> Output

 

I used intel's Color Plane Sequencer to separate RGB and A. I was not able to use this same component to combine YCbCr with Alpha since YCbCr is in 4:2:2 and has different bit sizes for each color space. I made the custom combiner component to handle this.

Y(8 bits) Cb(4 bits) Cr(4 bits)

 

I have not been getting adequate results so I learnt how to use the Signal Tap Logic Analyzer to debug. The custom combiner I made does not seem to be outputting any data. I will provide my new chroma keyer and custom combiner component. The chroma keyer signals and ports are still the same names as before because I haven't manually changed them yet but the logic has been changed to handle RGB.

 

The goal is to chroma key RGB 4:4:4 video data to remove the green residuals leftover from chroma keying YCbCr 4:2:2, but the output still needs to receive YCbCr w/ A 4:2:2

 

Thanks,

Evan

0 Kudos
khtan
Employee
2,250 Views

Hi Evan,

Thanks for the information. Let me check with the team and get back to you.

 

Thanks

Regards

Kian

0 Kudos
khtan
Employee
2,183 Views

Hi Evan, 

Our video team provided some links and commented that 

 

"If the customer has different bit sizes for each color space they will run into trouble.  They need to pad or crop color planes so that the IPs see the same number of planes and then use the IPs.  If they want to consider adding protocol convertors so they can use the VVP IPs, the bits per color sample does this. "

 

See :

 

https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/avalon-streaming-video-protocol-versus.html

 

https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/protocol-converter-ip-parameters.html

 

https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/bits-per-color-sample-adapter-ip-parameters.html

 

https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/color-plane-manager.html

 

https://www.intel.com/content/www/us/en/docs/programmable/683329/24-2/about-the-chroma-resampler-ip.html

 

Thanks

Regards

Kian

0 Kudos
khtan
Employee
2,095 Views

Hi Evan,

Just checking up on this case status whether you have any updates?

 

Thanks

Regards

Kian

0 Kudos
khtan
Employee
1,935 Views

Hi Evan,

I would like to inquire if there have been any updates.

Looking forward to hearing back from you soon.

Regards,
Kian

0 Kudos
EvanHamiltonJAVS
1,925 Views
Hey Kian,

I’m sorry for the late response, we are in the later stages of development for a different project and have been prioritizing it as of late. I haven’t been able to work much more on this yet since my last update. This project is still important to us, and I will let you know as soon as we start back up. I think the VVP IP components could help a lot and save me from making a custom converter from scratch.

Thanks,
Evan
0 Kudos
Reply