Tone Mapping Operator IP Malfunctioning

[dscully]

Hello,

I am currently evaluating the Tone Mapping Operator IP from the Video and Vision Processing Suite for integration into our video camera product. The issue is that when attempting to enable the tone mapping operation the core either stops producing frames or if I use the region of interest (ROI) function the video displays but the ROI area appears solid black regardless of volume and threshold levels.

Setup

Quartus Version: 23.4.0 Pro Edition

IP Configuration:

• OpenCore Evaluation Mode (time limted SOF loaded with nios2-configure)
• Number of pixels in parallel: 4
• Number of color planes: 3
• Bits per color sample: 12
• internal_cpu_clock/external_cpu_clock frequency: 150Mhz
• vid_clock frequency: 172Mhz
• Resets are associated with their clocks

Software Configuration (running on Nios II Core):

On startup, intel_vvp_tmo_init_instance is called and succeeds with a zero return code.

If debug prints are enabled (INTEL_VVP_TMO_ENABLE_LOGGING)
then the following prints out:

TILES:          4
BPP:            12
PPC:            4

intel_vvp_tmo_set_bypass is set to 1, and intel_vvp_tmo_set_resolution sets 1920x1080

The video pipeline starts at this point and frames pass normally.

Dumping the results of all get get functions of the TMO API results in:

get_bypass: 1
get_volume: 0
get_threshold: 0
get_debug: 15
get_roi_en: 0
get_roi_swap: 0
get_roi_horizontal_x0: 0
get_roi_horizontal_x1: 0
get_roi_vertical_y0: 0
get_roi_vertical_y1: 0
get_pip: 4
get_resolution_width: 1920
get_resolution_height: 1080

The debug register isn't directly documented as far as I can tell, but the "Software Driver Example" section gives the impression that 15 means all four rows are "Missing data collection per tile". I then clear the status with intel_vvp_tmo_set_debug(&tmo_instance, 15) which is reflected when I dump the registers again.

If I then set a region of interest using these commands:

intel_vvp_tmo_set_roi_horizontal_x0x1(&tmo_instance, 200, 400)
intel_vvp_tmo_set_roi_vertical_y0y1(&tmo_instance, 200, 400)

I see a black bar appear over the video feed.

I tried many combinations of volume/threshold and the pattern is unchanging. On some attempts writing those commands causes the processor to hang. If I try to disable bypass without using ROI, the core stops producing frames immediately.

Here is a direct dump of all registers for the ROI mode:

TMO_HW_INFO_0: 0x14A0043C
TMO_HW_INFO_1: 0x16565156
TMO_HW_INFO_2: 0x00094014
TMO_FLOW_CTL: 0x00000002
TMO_ACT_GEOM: 0x07800438
TMO_VOL_CTL: 0x00000000
TMO_CENTRE_TILE_WIDTH: 0x01E00000
TMO_CORNER_TILE_NORM: 0x0000102E
TMO_X_TILE_NORM: 0x00000817
TMO_Y_TILE_NORM: 0x00000817
TMO_CENTRE_TILE_NORM: 0x0000040B
TMO_ACT_TILE_DIM: 0x0078010E
TMO_BORDER_INFO_00: 0x007600EE
TMO_BORDER_INFO_01: 0x016601DF
TMO_BORDER_INFO_02: 0x01DE010E
TMO_BORDER_INFO_03: 0x010D021C
TMO_BORDER_INFO_04: 0x021B032A
TMO_BORDER_INFO_05: 0x03290438
TMO_BORDER_INFO_06: 0x043700F0
TMO_BORDER_INFO_07: 0x02D004B0
TMO_BORDER_INFO_08: 0x06900087
TMO_BORDER_INFO_09: 0x019502A3
TMO_BORDER_INFO_10: 0x043703B1
TMO_BORDER_INFO_11: 0x00EF02CF
TMO_BORDER_INFO_12: 0x04AF068F
TMO_BORDER_INFO_13: 0x00EE02CE
TMO_BORDER_INFO_14: 0x04AE068E
TMO_BORDER_INFO_15: 0x00ED02CD
TMO_BORDER_INFO_16: 0x04AD068D
TMO_BORDER_INFO_17: 0x00EC02CC
TMO_BORDER_INFO_18: 0x04AC068C
TMO_BORDER_INFO_19: 0x077F077E
TMO_BORDER_INFO_20: 0x077D077C
TMO_BORDER_INFO_21: 0x00860194
TMO_BORDER_INFO_22: 0x02A203B0
TMO_ROI_X0X1: 0x00C80190
TMO_ROI_Y0Y1: 0x00C80190

Are there any problems with this configuration that is preventing this from working?

Thank you,

David

[Wincent_Altera]

Hi,

May I know what device that you are targeting on this project ?

There is some idea out of my mind where we can try to narrow down the issue further bit

1. before TMO insert a gain on R only (e.g., scale R up). If ROI stays black but the rest of the frame changes, the input is RGB; if nothing changes, the source into TMO may be packed/zeroes.
2. set ROI to almost full frame and enable it. If black goes away → ROI alignment/width issue.
3. Do you try other lower/higher resolution rather than 2k ? is this issue happen in other resolution ?

Regards,

Wincent_Altera

[dscully]

Hello Wincent,

Thank you for the quick response. The target device is 10AX115H2F34E2SG.

1. I adjusted the red gain via a color space converter and the ROI region remained black while the color bars changed.
2. When I attempt to use larger ROI values typically the function call to intel_vvp_tmo_set_roi_horizontal_x0x1 or intel_vvp_tmo_set_roi_vertical_y0y1 will hang the nios2 processor. On one attempt I was able to set vertical 0->1060 and horizontal to 0->1900. The black area still appeared with a small bit of the test pattern visible.
3. I also happened to try a 800x800 camera feed and the results were the same.

Color Bar with Red Gain Set to ~8x:

800x800 Video:

[Wincent_Altera]

HI @dscully ,

Thanks for trying that, I just try to understand better

intel_vvp_tmo_set_bypass is set to 1, and intel_vvp_tmo_set_resolution sets 1920x1080

The video pipeline starts at this point and frames pass normally.
>> means by default it is working well (without black bar)
>> Am I get this correct ?

If I then set a region of interest using these commands:

intel_vvp_tmo_set_roi_horizontal_x0x1(&tmo_instance, 200, 400)
intel_vvp_tmo_set_roi_vertical_y0y1(&tmo_instance, 200, 400)

 >> The things happen ONLY when you try to set the ROI ?


Regards,
Wincent

[dscully]

Hi Wincent,

If I set the resolution with intel_vvp_tmo_set_resolution and then try to exit bypass mode the component stops producing frames. In ROI mode, the component still produces frames but shows the black box as described above.

Thanks,

David

[Wincent_Altera]

Hi @dscully  David,

If you not perform that , you wont see the black box ? Am I get this correct ?

If I set the resolution with intel_vvp_tmo_set_resolution and then try to exit bypass mode the component stops producing frames.
>> just try to understand why you not use the default setting but trying out this ?
>> Is that part of your project requirement ?
>> If I understand correctly from you, by default on VVP design this "black box" issue is not observed in 2k or 800x800 .
But once you make the change as your previous reply the black box came out , am I understand this correctly ?

Regards,
Wincent

[dscully]

Hi Wincent,

If I don't use ROI mode I don't see a black box. However, I expect in non-ROI mode for the component to function and not freeze the video pipeline when leaving bypass mode. I don't know what you mean by not using the "default settings". I am following the example from the documentation which clearly sets the resolution with intel_vvp_tmo_set_resolution:

Software Driver Example

By "default" this design does not perform its intended functionality regardless of resolution in ROI or non-ROI mode.

Thanks,

David

[Wincent_Altera]

Hi @dscully David,

Thanks for the clarification, now I am clear about your situation.
Please allow me to have sometime to check on this, it might take sometime.
Will get back to you as soon as possible.

Regards,

Wincent

[Wincent_Altera]

Hi @dscully David,

Based on the information you provided so far, my initial guess would be that most likely the ROI has not been properly defined in the SW App.

BUT , Before jumping to conclusions, and assuming the TMO has been initialize correctly, we would like a Yes or No on the following questions.

1. When TMO is in Bypass mode (reg_0x154 = 0x1), does the video passthrough without any artifacts?
2. When TMO is in Processing mode (reg_0x154 = 0x0),   does the video passthrough without any artifacts?
3. When TMO is in ROI mode (reg_0x154 = 0x2 or 0x6),   does the video passthrough without any artifacts?

At the meantime, I will email you privately some Genlock demo that uses the TMO IP, and from the jtag-command you can modify a horizontal slider based on TMO ROI registers. (Let me know if you not receive it)

Regards,

Wincent

[dscully]

Hi Wincent,

1. Yes, the video passes through without artifacts.

2. No, as soon as you switch from Bypass to Processing mode the IP stops producing frames.

3. No, black boxes appear based on the programmed ROI. The non-black box areas appears to be passed through.

To clarify, my earliest attempts did not use the ROI at all. I followed the "Software Driver Example". However, leaving bypass causes the core to stop producing frames immediately. I mentioned ROI because it produced different results than a IP freeze and thought it might be helpful for debug. I would be happy to see the IP function normally without ROI (full frame processing).

I received the design. I believe I have a copy of this design from a previous inquiry but I will verify tomorrow.

Thank you,

David

[dscully]

Hi Wincent,

I reviewed the example code you sent me. The setup of the core is here:

 intel_vvp_tmo_instance tmo;
 intel_vvp_tmo_base_t TMO_PIPE_0_TMO_BASE = reinterpret_cast<intel_vvp_tmo_base_t>(VIDEO_PROC_PIPE_0_TMO_BASE);
 intel_vvp_tmo_init_instance(&tmo, TMO_PIPE_0_TMO_BASE);
 intel_vvp_tmo_set_bypass(&tmo, 1);
 intel_vvp_tmo_set_volume(&tmo, 45);
 intel_vvp_tmo_set_threshold(&tmo, 4500);
 intel_vvp_tmo_set_resolution(&tmo, PIPE_DIM.x, PIPE_DIM.y);

I changed my setup to exactly match this order:

intel_vvp_tmo_init_instance(&cam_tmo, (intel_vvp_core_base)VVP_MAIN_PATH_VVP_MAIN_INTEL_TONEMAP_BASE);
intel_vvp_tmo_set_bypass(&cam_tmo, 1);
intel_vvp_tmo_set_volume(&cam_tmo, 45);
intel_vvp_tmo_set_threshold(&cam_tmo, 4500);
intel_vvp_tmo_set_resolution(&cam_tmo, 1920, 1080);

I verified that frames were flowing through the IP by checking the field count registers of the VVP IPs (IMG_INFO_FIELD_COUNT) before the TMO IP and after. The resolution registers (IMG_INFO_WIDTH/IMG_INFO_HEIGHT) for these IPs also match the set resolution.

I then ran a terminal command that calls intel_vvp_tmo_set_bypass(&cam_tmo, 0);

Sometimes the processor will hang on this write. Other times it does not freeze but the other IPs report a non-changing field count.

As you can see, there are issues even if not using ROI.

Best,

David

[Wincent_Altera]

Hi David,

Sometimes the processor will hang on this write. Other times it does not freeze but the other IPs report a non-changing field count.
>> Can I know how often this happen ?
>> What is the other IPs report non-charging field count ? Can you please at least give me few of the affected IP and the error message (if have) ? so that I can better visualize what you seeing just now.

As you can see, there are issues even if not using ROI.
>> Are you using the same device OPN as the design example ?
>> can you please use the design without modified anything ? Just to help us to narrow down the problem.
>> If you still seeing the same error from the design example , then we shall move our direction to the HW settings. Else we will focus on what is the minor different between yours design with our design provided.

Regards,
Wincent_Altera

[dscully]

Hi Wincent,

I have discovered that part of the issue is that this core requires the Lite variant of the video protocol and I was passing the Full variant to the core. I have since fixed this by adding protocol converters on both ends of the IP.

However, I am now facing new issues related to the resolution of the incoming video.

I have four sources of 60FPS video with varying resolutions:
800x800
1944x1092
3268x1840
1920x1080

When the video source changes, I call intel_vvp_tmo_set_resolution and set the lite to full protocol converter after the IP to the same resolution.

For 800x800 and 1920x1080 resolutions the tone mapping operation appears to apply correctly.

However for 1944x1092 and 3268x1840 the IP appears to be incorrectly applying the operation.

Processing Mode 3268x1840

Bypass Mode 3268x1840

Does this IP have undocumented restrictions on input resolution? The only restriction in the documentation I see is "Video resolutions up to 8192x4320 at 60 fps or 8192x8192 at 30 fps". Does the width or height need to be a multiple of the pixels in parallel? Does the ratio of width and height matter?

Thanks,

David

[Wincent_Altera]

Hi David,

Does this IP have undocumented restrictions on input resolution?
>> Are you referring to the provided design example or IP itself ?
>> if the design example, it only support 
>> 

The only restriction in the documentation I see is "Video resolutions up to 8192x4320 at 60 fps or 8192x8192 at 30 fps".
>> IF the TMO IP itself , the answer is YES
Does the width or height need to be a multiple of the pixels in parallel? Does the ratio of width and height matter?
>> Theoretically you need it, But to be honest I never try this implement before. 
>> But based on my understanding The TMO (Tone Mapping Operator) IP core processes a set number of pixels in parallel per clock cycle, often referred to as pixels_in_parallel. For efficient hardware utilization and to avoid partial processing at the end of each row, the frame width must be evenly divisible by the pixels_in_parallel parameter.
>> You can refer to 38. Pixels in Parallel Converter IP

Regards,

Wincent_Altera

[dscully]

Hi Wincent,

Can you confirm with the IP designers if there are other restrictions? Why does the IP malfunction if 1944x1092
3268x1840 are used since both widths are divisible by 4? Reminder that 800x800 and 1920x1080 function correctly in 4 pixels-in-parallel mode.

Thank you,

David

[Wincent_Altera]

Hi @dscully ,

Please allow me to check , it may take sometime .
Will get back to you immediately once I get the response.

Regards,
Wincent

[Wincent_Altera · Accepted solution]

Hi @dscully  David,

Does this IP have undocumented restrictions on input resolution? 
>> Yes, there is restrictions on the input resolution.

For the input width, it needs to be multiple of the pixels in parallels (PIP) and the number of internal tiles (it is fixed as 4)

• So, if PIP = 4 and Number of tiles = 4, then the width needs to be multiple of 4x4 = 16
• For the given two widths you could see that condition is not met, 1944/16 = 121.5 and 3268/16=204.25
• I suggest your to try: 1952 for the first case,
  • and 3280 for the second case, then if need be, you could use a clipper to put the width back to your desired width.
• For the input height, it needs to be multiple of the number of internal tiles (it is fixed as 4)
  • In this case, your current values are both fine.

Regards,
Wincent_Altera

[dscully]

Hi Wincent,

Thank you for this information. I used a upstream clipper to experimentally arrive at a similar ruleset. My findings are slightly different as the rule appears to be:

Resolution must be (N*8*PIP) x (M*8), where N and M are integers

For PIP=4: N*32 x M*8

For example:

Testing height rule:
1920x1080 (div by 4,8): Working
1920x1076 (div by 4): Looks ok at first, but image flickers on certain scenes
1920x1072 (div by 4,8): Working

Testing width rule:

1920x1080 (div by 16,32): Working
1904x1080 (div by 16): Creates 4 vertical white lines spaced across the screen
1888x1080 (div by 16,32): Working

I am not sure why I need double your recommendations but with this rule I can proceed evaluating the TMO IP.

I highly recommend that this rule be stated in the next revision of the "Video and Vision Processing Suite IP User Guide" so that others do not run into this issue. I have a feeling that there is a similar rule needed for the Warp IP as I had issues using it while warping at 798x798 image until I switched to 800x800.

Thank you for your support,
David

[Wincent_Altera]

Hi David,

Thanks for experiment this and providing such valuable information on how your resolve the problem.
I will feedback this to our IP designer, thanks again

Regards,
Wincent