GPU engine reset / job timeout under tensor-parallel LLM inference on dual Arc Pro B70 (Battlemage)

[triarii]

I've been running llama.cpp's SYCL backend, built from source myself (commit d2a818231, 2026-07-26) against Intel oneAPI 2026.1 (DPC++/C++ compiler 2026.1.1, build 2026.1.1.20260724), with Level Zero / compute-runtime libze-intel-gpu1 / intel-opencl-icd 26.22.38646.7 and IGC 2.36.5. Tensor-parallel inference (--split-mode tensor) across my two GPUs worked fine for a while, but broke sometime in May and has been unusable since.

Under sustained or concurrent inference load with --split-mode tensor, one GPU's engine faults and resets mid-job, logging Engine memory CAT error [5]: class=bcs (also seen as class=ccs), followed by Timedout job and Engine reset. It's consistently the GPU at PCI 07:00.0, though guc_id varies per occurrence. The SYCL/Level-Zero runtime never observes the reset and busy-spins in sched_yield() forever, so the llama-server worker pins a CPU core at 100% with zero progress. The process itself doesn't crash or report an error, only the affected request hangs indefinitely and needs a SIGKILL to clear.

In my testing, two or more concurrent large-prompt requests reliably reproduce it under tensor-split, even with fully non-overlapping prompts, which rules out KV-cache-reuse/prefix-matching as the cause. Plain concurrent GPU load is enough on its own. --split-mode layer (layer-parallel instead of tensor-parallel) has not reproduced the fault, but obviously gives up the throughput/latency benefit of running tensor-parallel across both cards. On one occasion the underlying condition triggered a full, unattended power-cycle of the entire machine rather than just a hung process. The BMC SEL showed no button-press event, ruling out physical intervention, and there was no forensic trail anywhere beyond the power-state transition itself.

I've confirmed this is a dupe of gitlab.freedesktop.org/drm/xe/kernel issue #8390, which traces the root cause to the GuC firmware's hardcoded 640ms preemption timeout on the ccs/bcs engine being exceeded under sustained or irregular decode work. It isn't tunable at runtime in native (non-SR-IOV) mode. The preempt_timeout_us debugfs knob only affects SR-IOV VF provisioning. I also tried xe.force_execlist=1 to bypass GuC entirely, but that breaks GPU POST outright on this hardware. As a sanity check I tried the Vulkan backend instead of SYCL: Vulkan correctly returns VK_ERROR_DEVICE_LOST instead of hanging under the same load, so the failure is at least detectable there, but it's 2.5 to 4x slower on token generation under tensor-split due to the per-token cross-GPU sync it requires, so it isn't a practical substitute. I also tried mainline kernel 7.2.0-070200rc5 hoping for a newer Xe driver fix, but that testing got blocked by an unrelated compute-runtime regression (since resolved) before I could get a clean read on whether it affects this bug at all. One workaround mentioned elsewhere, SYCL_UR_USE_LEVEL_ZERO_V2=0 (forcing the legacy L0 adapter), I'd actively steer people away from. On this hardware it produced a full hard machine lockup requiring physical power-cycle, which is worse than the bug itself.

Right now production is stable running --split-mode layer, which avoids triggering the fault, but I'd like to get back to tensor-parallel since it used to work and I'd rather not give up the performance it offers across both GPUs. Has anyone else gotten tensor-parallel/tensor-split stable on dual B70 (or other Battlemage multi-GPU setups) without hitting this, and does anyone know of a GuC firmware update, kernel patch, or driver-side mitigation beyond xe.wedged_mode=0 (which makes the reset auto-recoverable but doesn't prevent it)? I'm happy to provide more logs or dmesg captures, this is a well isolated, repeatable trigger on my end. For reference: the xe kernel tracker issue is at https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/8390, the llama.cpp-side tracking issue is ggml-org/llama.cpp#24810, and there's a similar report on vLLM with the same dual Arc Pro B70 / TP=2 setup at vllm-project/vllm#41663.

I'd also like to put this directly to Intel, since this seems like a driver/firmware-level issue rather than something the llama.cpp or vLLM communities can fix on their own. Is there any official guidance on working around the GuC's 640ms preemption timeout on Battlemage short of avoiding tensor-split entirely, and is a fix (whether a GuC firmware update, a kernel driver change, or an exposed tuning knob for the timeout in native, non-SR-IOV mode) expected or already planned? Given that this worked previously and regressed, I'd also appreciate knowing whether this is a known regression internally, and if so, roughly when a fix might land.

For hardware: this is a dual Intel Arc Pro B70 (Battlemage G31) setup, PCI 07:00.0 and 43:00.0, on an AMD EPYC 7232P host, running Ubuntu 26.04 LTS (Resolute Raccoon), kernel 7.0.0-30-generic, Xe driver xe 1.1.0 with GuC firmware bmg_guc_70.bin v70.58.0, HuC bmg_huc.bin v8.2.10, and DMC bmg_dmc.bin v2.6. No PCIe/riser signal issues observed, AER is clean.

Hardware: Dual Intel Arc Pro B70 (Battlemage G31),, AMD EPYC 7002 host

Software:
- OS: Ubuntu 26.04 LTS
- Kernel: 7.0.0-30-generic (stock Ubuntu 26.04 kernel)
- Xe driver: xe 1.1.0, GuC firmware bmg_guc_70.bin v70.58.0, HuC bmg_huc.bin v8.2.10, DMC bmg_dmc.bin v2.6
- Intel oneAPI: 2026.1 (DPC++/C++ compiler 2026.1.1, build 2026.1.1.20260724)
- Level Zero / compute-runtime: libze-intel-gpu1 / intel-opencl-icd 26.22.38646.7, IGC 2.36.5
- Inference engine: llama.cpp, SYCL backend, commit d2a818231 (2026-07-26), --split-mode tensor

[Vaultia]

Same hardware here — dual B70 (BMG-G31) — but a very different host: consumer platform, ASUS ProArt Z890-Creator, Core Ultra 7 265KF. Nice writeup, the isolation work is more thorough than most bug reports I've read.

Before I offer anything that might not apply: does your model fit within a single card's 32GB, or do you need the full 64GB?

Asking because it determines whether the tradeoff you're describing is actually forced. If you need 64GB, layer-split is mandatory and you're stuck choosing between it and tensor-split. If the model does fit on one card, there's a third option worth considering that I don't see mentioned — but it's moot in the first case, so I'd rather know before going into it.

One observation either way. Your control comparison changes two variables at once, not one: layer-split removes cross-GPU memory access, but it also removes simultaneous load on both cards, since it's sequential by design. So "layer-split is stable" is consistent with your GuC preemption theory, but it's equally consistent with the fault being tied to both cards drawing hard at the same time. Those need separating before the firmware conclusion holds.

I run the missing arm of that comparison in production — both cards under concurrent load, zero cross-GPU traffic — and haven't seen this. Different platform, so not conclusive for you, but it's a data point that the combination isn't inherently fatal on Battlemage.

[triarii]

>So "layer-split is stable" is consistent with your GuC preemption theory, but it's equally consistent with the fault being tied to both cards drawing hard at the same time. Those need separating before the firmware conclusion holds.

Interesting point, I had not though of that. I have been just chomping through any tests I could think of and have had a hard time keeping track. I'll try setting that up.

As for model size, I use Qwen Models for coding agents like open code. these are very context heavy. I find agents need about 70k context as bare minimum for useful coding session, generally more. the Q8 models are also much better than the Q5 or Q6 in my testing. current Qwen 3.8 Q8 is 28 - 31.5 GB so two cards is needed. 

[Vaultia]

That answers the sizing question — Q8 at 28-31.5GB plus 70k context is well past a single card, so split is genuinely forced for you. The pinning angle doesn't apply to your workload; ignore it.

Worth establishing first: my software stack is identical to yours. Ubuntu 26.04, kernel 7.0.0-30-generic, Xe driver, GuC 70.58.0, HuC 8.2.10, DMC 2.6. Different host — consumer ASUS ProArt Z890-Creator with a Core Ultra 7 265KF, cards at PCI 04:00.0 and 08:00.0 instead of your EPYC 7232P. Same firmware, same kernel, same driver, different platform.

I ran the missing arm of your comparison tonight, three configurations, checking dmesg after each.

First, two separate model instances pinned one per card via -sm none --main-gpu, zero cross-GPU traffic, both cards under concurrent load simultaneously. 8 requests in flight across the two servers, 173.6 t/s aggregate. Clean.

Second, Qwen3-Coder-Next 80B Q4_K_M (46GB) with --split-mode layer --tensor-split 50,50, genuinely spread across both cards, 4 concurrent requests at 512 tokens. Clean.

Third, closest to your regime: same 80B layer-split, two concurrent requests at ~25k prompt tokens each, distinct prompts with no shared prefix (verified in the logs — two full prefills, no cache reuse), server restarted beforehand to clear KV state. Prefill ran at 569 and 599 t/s, sustained cross-GPU work for 40+ seconds. Clean. No reset, no CAT error, no hung job, no runaway CPU spin.

So on identical firmware, none of these are sufficient to trigger it: concurrent load on both cards, cross-GPU layer-split under concurrency, or large-context prefill under concurrency. That separates the two variables your layer-split control was changing together. I'm at 25k rather than your 70k, so I can't claim to have covered your full regime.

Then I tried to reproduce with tensor-split, and that's where it got interesting. It won't run here at all — segfault during model init, before any tensors are allocated. The dmesg line immediately preceding it:

xe 0000:08:00.0: cannot be used for peer-to-peer DMA as the client and provider (0000:04:00.0) do not share an upstream bridge or whitelisted host bridge

followed by a segfault at address 1 inside libggml-sycl.so. Peer-to-peer DMA between the two cards is refused by the kernel here, tensor-split needs it, and the SYCL path doesn't check the failure before dereferencing.

Checking the topology confirms it — the two cards sit under separate root ports with no common upstream bridge:

/sys/devices/pci0000:00/0000:00:06.0/0000:02:00.0/0000:03:01.0/0000:04:00.0
/sys/devices/pci0000:00/0000:00:06.3/0000:06:00.0/0000:07:01.0/0000:08:00.0

The paths diverge at the first bridge. Your EPYC 7232P is a server platform with all PCIe lanes off the CPU and a common root complex, so P2P DMA works there. That's likely why tensor-split starts on your machine and dies on mine.

And that may be worth more to your diagnosis than my null result. If P2P DMA is functional on your setup, the code path failing under load is direct card-to-card memory transfer. That fits "Engine memory CAT error" better than a preemption timeout does — a CAT error is a catastrophic memory access fault, and in your logs it appears first, with the reset and the timeout following it. Worth considering whether multi-root USM handling on Battlemage is a more promising place to look than the GuC's 640ms timeout.

One other thing from the 25k test that may matter for your actual workload. The two slots did not prefill in parallel in any meaningful sense. Slot 0 prefilled at full speed, peaking near 685 t/s, while slot 1 crawled at 71 t/s and only climbed as slot 0 finished. Then slot 0 decoded at 6.28 t/s while slot 1 completed its own prefill — versus 22.20 t/s once slot 1 had the cards to itself. A 3.5x degradation on concurrent decode, purely from the other slot's prefill. For contrast, with short prompts: 12B pinned to one card scales 4.5x from single-session to 6 slots (32 to 144 t/s); 80B layer-split with short prompts scales 1.26x. At 25k prompts, concurrency stops helping anyone.

If your coding agents run at 70k with multiple requests in flight, that's the regime you're in, and the cost looks structural rather than something tensor-split would fix. Might be worth measuring what your layer-split setup actually delivers at concurrency 2 with realistic prompt sizes before investing more — the gain you're chasing could be smaller than the one you're already losing.

Last thing, which follows from the stacks matching. If firmware, kernel and driver are identical, whatever differs is hardware. You list GuC, HuC and DMC to the exact version but never mention the PSU. What are the two cards running on, and what are they drawing under sustained tensor-split? I ask because of the unattended full power-cycle: no GPU firmware bug cuts system power. An engine reset is a contained per-engine recovery, and even a fully wedged GPU doesn't drop the host. A spontaneous power-cycle with no forensic trail beyond the power-state transition is what a supply-side protection tripping looks like — OCP on a transient, VRM, or thermal. The fault landing consistently on 07:00.0 also fits a physical asymmetry better than a firmware timeout, which shouldn't be that deterministic about which card falls over.