Original title: Is memory the moat?
Article
Open model size and context demands are driving deployments from closed models toward large open models such as Kimi K3, but the 2.8T-parameter footprint and 1M-token context make memory capacity the central constraint. The post reports that AMD MI355X, with 288GB HBM and lower GPU-hour pricing, can outperform Nvidia B200 in aggregate throughput and beat B300 on performance per dollar despite lower single-stream speed and a slower prefill baseline in raw terms. The team measured about 952 tokens/s per node on a 1,024-in/400-out benchmark and 118 tok/s per stream, versus 498 tok/s and 90 tok/s on a TP16 B200 setup, while noting B300 still has higher aggregate throughput at higher cost. The benchmark setup includes a B200 limitation: two-node cross-node communication on decode dampens its efficiency, so MI355X’s single-node memory profile changes the tradeoff. Most gains came from targeted fixes in the ROCm stack rather than heavy custom kernel engineering: a missing top-k renorm symbol in sglang broke speculative decode and was patched with a small PyTorch function, raising single-stream and aggregate throughput significantly. Prefill performance was then optimized by aligning attention head counts to use AITER’s fast MLA path, turning a multi-minute bottleneck into a 2–3x improvement in time-to-first-token behavior. The post concludes that near-SOTA performance per dollar on AMD is now achievable with fewer framework patches than previous frontier ports. Readers point to the simplicity of some fixes as a reason to question whether benchmark correctness was preserved and whether those optimizations were fully verified.
A participant questioned whether the setup and benchmarking were fully coherent, arguing that AI-assisted tuning claims like the head-count padding trick can introduce subtle correctness failures even when throughput improves. The concern centered on a prior experience with an assistant-generated optimization that made a small model incoherent, and the critic asks for evidence that Kimi K3 outputs remain faithful after the MI355X tuning steps. The same comment criticizes the post for insufficiently scrutinizing edge-case output quality, especially around the prefill-path changes, and frames this as a broader demand for rigor. Other remarks describe the writeup as "AI slop," implying weak technical confidence despite concrete numbers, and suggest that benchmark credibility depends on reproducing correctness checks, not only throughput metrics.