Re: Tone Mapping Operator IP Malfunctioning

dscully · ‎08-29-2025

Hello,

I am currently evaluating the Tone Mapping Operator IP from the Video and Vision Processing Suite for integration into our video camera product. The issue is that when attempting to enable the tone mapping operation the core either stops producing frames or if I use the region of interest (ROI) function the video displays but the ROI area appears solid black regardless of volume and threshold levels.

Setup

Quartus Version: 23.4.0 Pro Edition

IP Configuration:

OpenCore Evaluation Mode (time limted SOF loaded with nios2-configure)
Number of pixels in parallel: 4
Number of color planes: 3
Bits per color sample: 12
internal_cpu_clock/external_cpu_clock frequency: 150Mhz
vid_clock frequency: 172Mhz
Resets are associated with their clocks

Software Configuration (running on Nios II Core):

On startup, intel_vvp_tmo_init_instance is called and succeeds with a zero return code.

If debug prints are enabled (INTEL_VVP_TMO_ENABLE_LOGGING)
then the following prints out:

TILES:          4
BPP:            12
PPC:            4

intel_vvp_tmo_set_bypass is set to 1, and intel_vvp_tmo_set_resolution sets 1920x1080

The video pipeline starts at this point and frames pass normally.

Dumping the results of all get get functions of the TMO API results in:

get_bypass: 1
get_volume: 0
get_threshold: 0
get_debug: 15
get_roi_en: 0
get_roi_swap: 0
get_roi_horizontal_x0: 0
get_roi_horizontal_x1: 0
get_roi_vertical_y0: 0
get_roi_vertical_y1: 0
get_pip: 4
get_resolution_width: 1920
get_resolution_height: 1080

The debug register isn't directly documented as far as I can tell, but the "Software Driver Example" section gives the impression that 15 means all four rows are "Missing data collection per tile". I then clear the status with intel_vvp_tmo_set_debug(&tmo_instance, 15) which is reflected when I dump the registers again.

If I then set a region of interest using these commands:

intel_vvp_tmo_set_roi_horizontal_x0x1(&tmo_instance, 200, 400)
intel_vvp_tmo_set_roi_vertical_y0y1(&tmo_instance, 200, 400)

I see a black bar appear over the video feed.

I tried many combinations of volume/threshold and the pattern is unchanging. On some attempts writing those commands causes the processor to hang. If I try to disable bypass without using ROI, the core stops producing frames immediately.

Here is a direct dump of all registers for the ROI mode:

TMO_HW_INFO_0: 0x14A0043C
TMO_HW_INFO_1: 0x16565156
TMO_HW_INFO_2: 0x00094014
TMO_FLOW_CTL: 0x00000002
TMO_ACT_GEOM: 0x07800438
TMO_VOL_CTL: 0x00000000
TMO_CENTRE_TILE_WIDTH: 0x01E00000
TMO_CORNER_TILE_NORM: 0x0000102E
TMO_X_TILE_NORM: 0x00000817
TMO_Y_TILE_NORM: 0x00000817
TMO_CENTRE_TILE_NORM: 0x0000040B
TMO_ACT_TILE_DIM: 0x0078010E
TMO_BORDER_INFO_00: 0x007600EE
TMO_BORDER_INFO_01: 0x016601DF
TMO_BORDER_INFO_02: 0x01DE010E
TMO_BORDER_INFO_03: 0x010D021C
TMO_BORDER_INFO_04: 0x021B032A
TMO_BORDER_INFO_05: 0x03290438
TMO_BORDER_INFO_06: 0x043700F0
TMO_BORDER_INFO_07: 0x02D004B0
TMO_BORDER_INFO_08: 0x06900087
TMO_BORDER_INFO_09: 0x019502A3
TMO_BORDER_INFO_10: 0x043703B1
TMO_BORDER_INFO_11: 0x00EF02CF
TMO_BORDER_INFO_12: 0x04AF068F
TMO_BORDER_INFO_13: 0x00EE02CE
TMO_BORDER_INFO_14: 0x04AE068E
TMO_BORDER_INFO_15: 0x00ED02CD
TMO_BORDER_INFO_16: 0x04AD068D
TMO_BORDER_INFO_17: 0x00EC02CC
TMO_BORDER_INFO_18: 0x04AC068C
TMO_BORDER_INFO_19: 0x077F077E
TMO_BORDER_INFO_20: 0x077D077C
TMO_BORDER_INFO_21: 0x00860194
TMO_BORDER_INFO_22: 0x02A203B0
TMO_ROI_X0X1: 0x00C80190
TMO_ROI_Y0Y1: 0x00C80190

Are there any problems with this configuration that is preventing this from working?

Thank you,

David

Wincent_Altera · ‎09-01-2025

Hi,

May I know what device that you are targeting on this project ?

There is some idea out of my mind where we can try to narrow down the issue further bit

before TMO insert a gain on R only (e.g., scale R up). If ROI stays black but the rest of the frame changes, the input is RGB; if nothing changes, the source into TMO may be packed/zeroes.
set ROI to almost full frame and enable it. If black goes away → ROI alignment/width issue.
Do you try other lower/higher resolution rather than 2k ? is this issue happen in other resolution ?

Regards,

Wincent_Altera

dscully · ‎09-02-2025

Hello Wincent,

Thank you for the quick response. The target device is 10AX115H2F34E2SG.

I adjusted the red gain via a color space converter and the ROI region remained black while the color bars changed.
When I attempt to use larger ROI values typically the function call to intel_vvp_tmo_set_roi_horizontal_x0x1 or intel_vvp_tmo_set_roi_vertical_y0y1 will hang the nios2 processor. On one attempt I was able to set vertical 0->1060 and horizontal to 0->1900. The black area still appeared with a small bit of the test pattern visible.
I also happened to try a 800x800 camera feed and the results were the same.

Color Bar with Red Gain Set to ~8x:

800x800 Video:

Wincent_Altera · ‎09-02-2025

HI @dscully ,

Thanks for trying that, I just try to understand better

intel_vvp_tmo_set_bypass is set to 1, and intel_vvp_tmo_set_resolution sets 1920x1080

The video pipeline starts at this point and frames pass normally.
>> means by default it is working well (without black bar)
>> Am I get this correct ?

If I then set a region of interest using these commands:

intel_vvp_tmo_set_roi_horizontal_x0x1(&tmo_instance, 200, 400)
intel_vvp_tmo_set_roi_vertical_y0y1(&tmo_instance, 200, 400)

>> The things happen ONLY when you try to set the ROI ?

Regards,
Wincent

dscully · ‎09-02-2025

Hi Wincent,

If I set the resolution with intel_vvp_tmo_set_resolution and then try to exit bypass mode the component stops producing frames. In ROI mode, the component still produces frames but shows the black box as described above.

Thanks,

David

Wincent_Altera · ‎09-02-2025

Hi @dscully David,

If you not perform that , you wont see the black box ? Am I get this correct ?

If I set the resolution with intel_vvp_tmo_set_resolution and then try to exit bypass mode the component stops producing frames.
>> just try to understand why you not use the default setting but trying out this ?
>> Is that part of your project requirement ?
>> If I understand correctly from you, by default on VVP design this "black box" issue is not observed in 2k or 800x800 .
But once you make the change as your previous reply the black box came out , am I understand this correctly ?

Regards,
Wincent

dscully · ‎09-03-2025

Hi Wincent,

If I don't use ROI mode I don't see a black box. However, I expect in non-ROI mode for the component to function and not freeze the video pipeline when leaving bypass mode. I don't know what you mean by not using the "default settings". I am following the example from the documentation which clearly sets the resolution with intel_vvp_tmo_set_resolution:

Software Driver Example

By "default" this design does not perform its intended functionality regardless of resolution in ROI or non-ROI mode.

Thanks,

David

Wincent_Altera · ‎09-03-2025

Hi @dscully David,

Thanks for the clarification, now I am clear about your situation.
Please allow me to have sometime to check on this, it might take sometime.
Will get back to you as soon as possible.

Regards,

Wincent

Wincent_Altera · ‎09-03-2025

Hi @dscully David,

Based on the information you provided so far, my initial guess would be that most likely the ROI has not been properly defined in the SW App.

BUT , Before jumping to conclusions, and assuming the TMO has been initialize correctly, we would like a Yes or No on the following questions.

When TMO is in Bypass mode (reg_0x154 = 0x1), does the video passthrough without any artifacts?
When TMO is in Processing mode (reg_0x154 = 0x0), does the video passthrough without any artifacts?
When TMO is in ROI mode (reg_0x154 = 0x2 or 0x6), does the video passthrough without any artifacts?

At the meantime, I will email you privately some Genlock demo that uses the TMO IP, and from the jtag-command you can modify a horizontal slider based on TMO ROI registers. (Let me know if you not receive it)

Regards,

Wincent

dscully · ‎09-03-2025

Hi Wincent,

1. Yes, the video passes through without artifacts.

2. No, as soon as you switch from Bypass to Processing mode the IP stops producing frames.

3. No, black boxes appear based on the programmed ROI. The non-black box areas appears to be passed through.

To clarify, my earliest attempts did not use the ROI at all. I followed the "Software Driver Example". However, leaving bypass causes the core to stop producing frames immediately. I mentioned ROI because it produced different results than a IP freeze and thought it might be helpful for debug. I would be happy to see the IP function normally without ROI (full frame processing).

I received the design. I believe I have a copy of this design from a previous inquiry but I will verify tomorrow.

Thank you,

David

dscully · ‎09-04-2025

Hi Wincent,

I reviewed the example code you sent me. The setup of the core is here:

 intel_vvp_tmo_instance tmo;
 intel_vvp_tmo_base_t TMO_PIPE_0_TMO_BASE = reinterpret_cast<intel_vvp_tmo_base_t>(VIDEO_PROC_PIPE_0_TMO_BASE);
 intel_vvp_tmo_init_instance(&tmo, TMO_PIPE_0_TMO_BASE);
 intel_vvp_tmo_set_bypass(&tmo, 1);
 intel_vvp_tmo_set_volume(&tmo, 45);
 intel_vvp_tmo_set_threshold(&tmo, 4500);
 intel_vvp_tmo_set_resolution(&tmo, PIPE_DIM.x, PIPE_DIM.y);

I changed my setup to exactly match this order:

intel_vvp_tmo_init_instance(&cam_tmo, (intel_vvp_core_base)VVP_MAIN_PATH_VVP_MAIN_INTEL_TONEMAP_BASE);
intel_vvp_tmo_set_bypass(&cam_tmo, 1);
intel_vvp_tmo_set_volume(&cam_tmo, 45);
intel_vvp_tmo_set_threshold(&cam_tmo, 4500);
intel_vvp_tmo_set_resolution(&cam_tmo, 1920, 1080);

I verified that frames were flowing through the IP by checking the field count registers of the VVP IPs (IMG_INFO_FIELD_COUNT) before the TMO IP and after. The resolution registers (IMG_INFO_WIDTH/IMG_INFO_HEIGHT) for these IPs also match the set resolution.

I then ran a terminal command that calls intel_vvp_tmo_set_bypass(&cam_tmo, 0);

Sometimes the processor will hang on this write. Other times it does not freeze but the other IPs report a non-changing field count.

As you can see, there are issues even if not using ROI.

Best,

David

dscully · ‎09-05-2025

Hi Wincent,

I have discovered that part of the issue is that this core requires the Lite variant of the video protocol and I was passing the Full variant to the core. I have since fixed this by adding protocol converters on both ends of the IP.

However, I am now facing new issues related to the resolution of the incoming video.

I have four sources of 60FPS video with varying resolutions:
800x800
1944x1092
3268x1840
1920x1080

When the video source changes, I call intel_vvp_tmo_set_resolution and set the lite to full protocol converter after the IP to the same resolution.

For 800x800 and 1920x1080 resolutions the tone mapping operation appears to apply correctly.

However for 1944x1092 and 3268x1840 the IP appears to be incorrectly applying the operation.

Processing Mode 3268x1840

Bypass Mode 3268x1840

Does this IP have undocumented restrictions on input resolution? The only restriction in the documentation I see is "Video resolutions up to 8192x4320 at 60 fps or 8192x8192 at 30 fps". Does the width or height need to be a multiple of the pixels in parallel? Does the ratio of width and height matter?

Thanks,

David