Running LLMs on fanless embedded devices — realistic limitations

We often get the question in the field, "Can we run an LLM on a small box?" We can start with the conclusion. However, expectations need to be adjusted.

Memory bandwidth is the real bottleneck

LLM inference speed is governed by memory bandwidth rather than compute (FLOPS). This is because the entire model parameters must be read from memory for every token generated.

Running a 7B model with 4-bit quantization (Q4_K_M) on an N97-class device takes about 4.5GB. With an 8GB model, practical use is difficult considering the OS and headroom.

So where is it used?

• Small models of 3B or less — classification, summarization, tagging
• Embedding-only — RAG search indexing
• Narrow tasks such as voice command recognition and anomaly detection

I do not recommend a configuration that puts a general-purpose chatbot in an on-site box. If responses are slow, users will eventually stop using it. Instead, automating narrow, repetitive tasks has a greater perceived effect.

cooling design precautions

Fanless cases dissipate heat through heat fins. LLM inference is a sustained load, so it generates more heat than a momentary load. You should run it for more than 30 minutes in an ambient temperature of 40℃ and check for throttling.

The TDP on the spec sheet is the instantaneous maximum. It is important to measure the clock that is actually maintained under sustained load.