GPU engine reset / job timeout under tensor-parallel LLM inference on dual Arc Pro B70 (Battlemage)

[triarii]

I've been running llama.cpp's SYCL backend, built from source myself (commit d2a818231, 2026-07-26) against Intel oneAPI 2026.1 (DPC++/C++ compiler 2026.1.1, build 2026.1.1.20260724), with Level Zero / compute-runtime libze-intel-gpu1 / intel-opencl-icd 26.22.38646.7 and IGC 2.36.5. Tensor-parallel inference (--split-mode tensor) across my two GPUs worked fine for a while, but broke sometime in May and has been unusable since.

Under sustained or concurrent inference load with --split-mode tensor, one GPU's engine faults and resets mid-job, logging Engine memory CAT error [5]: class=bcs (also seen as class=ccs), followed by Timedout job and Engine reset. It's consistently the GPU at PCI 07:00.0, though guc_id varies per occurrence. The SYCL/Level-Zero runtime never observes the reset and busy-spins in sched_yield() forever, so the llama-server worker pins a CPU core at 100% with zero progress. The process itself doesn't crash or report an error, only the affected request hangs indefinitely and needs a SIGKILL to clear.

In my testing, two or more concurrent large-prompt requests reliably reproduce it under tensor-split, even with fully non-overlapping prompts, which rules out KV-cache-reuse/prefix-matching as the cause. Plain concurrent GPU load is enough on its own. --split-mode layer (layer-parallel instead of tensor-parallel) has not reproduced the fault, but obviously gives up the throughput/latency benefit of running tensor-parallel across both cards. On one occasion the underlying condition triggered a full, unattended power-cycle of the entire machine rather than just a hung process. The BMC SEL showed no button-press event, ruling out physical intervention, and there was no forensic trail anywhere beyond the power-state transition itself.

I've confirmed this is a dupe of gitlab.freedesktop.org/drm/xe/kernel issue #8390, which traces the root cause to the GuC firmware's hardcoded 640ms preemption timeout on the ccs/bcs engine being exceeded under sustained or irregular decode work. It isn't tunable at runtime in native (non-SR-IOV) mode. The preempt_timeout_us debugfs knob only affects SR-IOV VF provisioning. I also tried xe.force_execlist=1 to bypass GuC entirely, but that breaks GPU POST outright on this hardware. As a sanity check I tried the Vulkan backend instead of SYCL: Vulkan correctly returns VK_ERROR_DEVICE_LOST instead of hanging under the same load, so the failure is at least detectable there, but it's 2.5 to 4x slower on token generation under tensor-split due to the per-token cross-GPU sync it requires, so it isn't a practical substitute. I also tried mainline kernel 7.2.0-070200rc5 hoping for a newer Xe driver fix, but that testing got blocked by an unrelated compute-runtime regression (since resolved) before I could get a clean read on whether it affects this bug at all. One workaround mentioned elsewhere, SYCL_UR_USE_LEVEL_ZERO_V2=0 (forcing the legacy L0 adapter), I'd actively steer people away from. On this hardware it produced a full hard machine lockup requiring physical power-cycle, which is worse than the bug itself.

Right now production is stable running --split-mode layer, which avoids triggering the fault, but I'd like to get back to tensor-parallel since it used to work and I'd rather not give up the performance it offers across both GPUs. Has anyone else gotten tensor-parallel/tensor-split stable on dual B70 (or other Battlemage multi-GPU setups) without hitting this, and does anyone know of a GuC firmware update, kernel patch, or driver-side mitigation beyond xe.wedged_mode=0 (which makes the reset auto-recoverable but doesn't prevent it)? I'm happy to provide more logs or dmesg captures, this is a well isolated, repeatable trigger on my end. For reference: the xe kernel tracker issue is at https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/8390, the llama.cpp-side tracking issue is ggml-org/llama.cpp#24810, and there's a similar report on vLLM with the same dual Arc Pro B70 / TP=2 setup at vllm-project/vllm#41663.

I'd also like to put this directly to Intel, since this seems like a driver/firmware-level issue rather than something the llama.cpp or vLLM communities can fix on their own. Is there any official guidance on working around the GuC's 640ms preemption timeout on Battlemage short of avoiding tensor-split entirely, and is a fix (whether a GuC firmware update, a kernel driver change, or an exposed tuning knob for the timeout in native, non-SR-IOV mode) expected or already planned? Given that this worked previously and regressed, I'd also appreciate knowing whether this is a known regression internally, and if so, roughly when a fix might land.

For hardware: this is a dual Intel Arc Pro B70 (Battlemage G31) setup, PCI 07:00.0 and 43:00.0, on an AMD EPYC 7232P host, running Ubuntu 26.04 LTS (Resolute Raccoon), kernel 7.0.0-30-generic, Xe driver xe 1.1.0 with GuC firmware bmg_guc_70.bin v70.58.0, HuC bmg_huc.bin v8.2.10, and DMC bmg_dmc.bin v2.6. No PCIe/riser signal issues observed, AER is clean.

Hardware: Dual Intel Arc Pro B70 (Battlemage G31),, AMD EPYC 7002 host

Software:
- OS: Ubuntu 26.04 LTS
- Kernel: 7.0.0-30-generic (stock Ubuntu 26.04 kernel)
- Xe driver: xe 1.1.0, GuC firmware bmg_guc_70.bin v70.58.0, HuC bmg_huc.bin v8.2.10, DMC bmg_dmc.bin v2.6
- Intel oneAPI: 2026.1 (DPC++/C++ compiler 2026.1.1, build 2026.1.1.20260724)
- Level Zero / compute-runtime: libze-intel-gpu1 / intel-opencl-icd 26.22.38646.7, IGC 2.36.5
- Inference engine: llama.cpp, SYCL backend, commit d2a818231 (2026-07-26), --split-mode tensor

[Vaultia]

Same hardware here — dual B70 (BMG-G31) — but a very different host: consumer platform, ASUS ProArt Z890-Creator, Core Ultra 7 265KF. Nice writeup, the isolation work is more thorough than most bug reports I've read.

Before I offer anything that might not apply: does your model fit within a single card's 32GB, or do you need the full 64GB?

Asking because it determines whether the tradeoff you're describing is actually forced. If you need 64GB, layer-split is mandatory and you're stuck choosing between it and tensor-split. If the model does fit on one card, there's a third option worth considering that I don't see mentioned — but it's moot in the first case, so I'd rather know before going into it.

One observation either way. Your control comparison changes two variables at once, not one: layer-split removes cross-GPU memory access, but it also removes simultaneous load on both cards, since it's sequential by design. So "layer-split is stable" is consistent with your GuC preemption theory, but it's equally consistent with the fault being tied to both cards drawing hard at the same time. Those need separating before the firmware conclusion holds.

I run the missing arm of that comparison in production — both cards under concurrent load, zero cross-GPU traffic — and haven't seen this. Different platform, so not conclusive for you, but it's a data point that the combination isn't inherently fatal on Battlemage.

[triarii]

>So "layer-split is stable" is consistent with your GuC preemption theory, but it's equally consistent with the fault being tied to both cards drawing hard at the same time. Those need separating before the firmware conclusion holds.

Interesting point, I had not though of that. I have been just chomping through any tests I could think of and have had a hard time keeping track. I'll try setting that up.

As for model size, I use Qwen Models for coding agents like open code. these are very context heavy. I find agents need about 70k context as bare minimum for useful coding session, generally more. the Q8 models are also much better than the Q5 or Q6 in my testing. current Qwen 3.8 Q8 is 28 - 31.5 GB so two cards is needed.