The Diffusion Guide · ch.6 · prompt engineering
🎨 Chapter 6 · Prompt engineering

Say it so the model sees it

Five chapters of knobs, patches, and pictures — and the humble text box was a precision instrument all along. This chapter is how the encoder actually hears you, and how to phrase things so it listens.

You've steered with knobs, patches, and pictures. The one tool you've used since page one and never actually studied is the text box — and it turns out to be the fussiest instrument in the app.

Not because prompting is magic. Because the thing reading your prompt isn't a person. It's an encoder with a short attention span and a hard word limit, and once you know how it hears, "prompt engineering" stops being incantations you copy off Reddit and becomes a handful of rules you can reason about.

1The distracted intern

Back in Chapter 1 you met CLIP: the text encoder that turns your prompt into a list of numbers before the U-Net ever sees it. Here's the part that matters for everything below. That encoder reads your prompt like a distracted intern — the first things you say land hardest, and after about seventy-five words it stops listening entirely.

Three consequences fall out of that, and they drive the whole chapter:

  • Order matters. Early words carry more weight than late ones. The subject you lead with dominates; the adjectives you trail off with are suggestions.
  • The window is finite. Roughly seventy-five tokens fit. Everything past that isn't down-weighted — it's simply cut, never read at all.
  • Geometry barely survives. "On the left" arrives as a vibe, not a coordinate. That's not a bug you can prompt around — it's exactly why Chapter 5 exists.

So flip your mental model. You're not writing for a reader who'll patiently parse your paragraph. You're writing for an encoder that skims the top and drops the bottom. Everything else here is how to write for that.

💡
An FP framing: the prompt is data before it's prose. It gets parsed into a fixed-length vector, and position in the input is part of the signal. Treat it like you'd treat an ordered List where the head is load-bearing — because that's what it becomes.

2Anatomy of a prompt

A prompt that works usually has the same skeleton, whether the author knew it or not. Plainly, in order:

  • Subject, with details — what the picture is of. "a weathered lighthouse on a cliff." Lead here.
  • Medium — photo, watercolor, oil painting, comic ink. What it's made of.
  • Style / artist references — minimalist, ornate, "in the style of…". The aesthetic.
  • Lighting — golden hour, overcast, neon night, rim light. Cheap and high-leverage (section 6).
  • Camera / composition — 85mm portrait, wide-angle, bird's-eye. Where the viewer stands.
  • Quality tags — "detailed", "masterpiece". Optional, and more optional than the internet thinks (section 7).

Recognize the last four? That's Chapter 4's style block — the reusable chunk you froze verbatim so every render shared a look. It was these slots, locked. The subject changes shot to shot; the garnish stays put.

This isn't a law. It's a checklist, and its real value is negative: it makes you notice the slots you left blank. Every unspecified slot is a decision you handed to the model — and it will decide, every time, with or without your input. Build a prompt slot by slot and watch each choice land:

Interactive · the prompt anatomy builder Fill the slots · flip the order
order

3Front-load what matters

Order isn't grammar here — it's priority. The encoder weights early tokens more, so whatever you lead with is what the image commits to hardest. The subject phrase belongs at the front; the aesthetic garnish belongs behind it.

You can hear the shift. a lighthouse, watercolor paints a lighthouse and happens to do it in watercolor. Flip it to watercolor, a lighthouse and you've told the model the watercolor is the point — now you'll get a wash of paint that contains, somewhere, a lighthouse-ish shape. Same words, different boss. (You just watched this in the builder's order toggle: style-first visibly shrinks the subject.)

Which gives you a diagnostic. If your renders keep nailing the style and mangling the subject, your subject is buried too deep — promote it to the front. Adding more adjectives won't help; moving the important words up will.

🎯
The rule in one line: lead with the noun you care most about. Everything after it competes for the encoder's fading attention — so spend the front of the prompt on what the picture is, not on how pretty it should be.

4The token budget

CLIP's window is about seventy-five tokens. Cross it and the extra words aren't quietly de-emphasized — they're truncated, dropped before the model ever reads them. This is the single most common self-inflicted bug in prompting: a gorgeous 120-word prompt whose final, most specific clauses were never seen. You keep adding detail; the model keeps ignoring it; both of you are technically doing your jobs.

Two things trip people up. First, tokens aren't words — common words are one token, but unusual or long ones get split into pieces, so "photorealistic" or "chiaroscuro" can cost two or three. Second, the cut is silent. Nothing warns you. Paste an overlong prompt below and watch where the encoder stops reading:

Interactive · the token window Edit the text · watch the budget
0 tokens75 · the budget

Token count is approximate — a rough stand-in for CLIP's real tokenizer (each word ≈ one token; long or unusual words count as two). Good enough to feel the ceiling.

The fix is a discipline, not a trick: trim ruthlessly, and prefer strong nouns to adjective chains. "a derelict lighthouse" beats "an old, weathered, crumbling, abandoned, decaying lighthouse" — it says more and costs a fifth of the budget. (One soft caveat: some newer UIs chain multiple 75-token windows so long prompts don't hard-truncate. You can't count on which yours does, so write as if the budget is real — it costs you nothing when it isn't.)

5Emphasis, when your UI supports it

Sometimes a word needs to shout. There's a community syntax for that, and many UIs speak it:

  • (word) — nudges the word's weight up a little.
  • (word:1.3) — sets an explicit weight; above 1 louder, below 1 quieter.
  • [word] — nudges it down.

Whether a given DiffusionBee build honors this depends on the version, so treat it as a bonus, not a foundation. The good news: the concept transfers everywhere, and the two universal fallbacks always work — reorder (move the word earlier) and repeat (say it twice). Emphasis syntax is just a dial for something you can already do by hand.

And it has a ceiling. Push a word past roughly 1.4 and it behaves exactly like Chapter 2's fried guidance: it stops being emphasis and starts being a takeover, dragging the whole image toward that one word. Drag the dial and watch "misty" go from ignored to natural to landscape-eating fog:

Interactive · the emphasis dial Drag from ignored to takeover
ignored natural dominant takeover

6Camera and lighting vocabulary

Here's the highest-leverage cheap trick in the whole chapter. The training data is saturated with photographic language — millions of captioned images tagged with lenses, times of day, and lighting setups. Speak that language and you get precise, repeatable control for free.

Reference · vocabulary that pays
Say…Get…Use it for
85mm portraitshallow depth of field, blurred backgroundflattering close portraits, a subject that pops
wide-anglea broad, slightly distorted fielddrama, landscapes, cramped interiors made grand
macroextreme close detailtexture, insects, tiny objects made huge
golden hourwarm, low, directional suncozy, cinematic warmth without lifting a finger
rim lighta glowing outline around the subjectseparating a figure from a dark background
overcastsoft, flat, even lightgentle portraits, no harsh shadows
low-keydark, moody, mostly shadownoir, tension, drama
rule of thirds / centeredoff-balance vs. dead-center framingdeciding where the eye lands
from below / bird's-eyea low heroic angle vs. a top-down mappower, scale, or a laid-out overview

Think of this as style control without a LoRA. It's free, stackable, and reversible: layer a lens, a time of day, and a framing in one prompt, and pull any of them back out with a keystroke — no downloads, no training, no gigabytes. Before you reach for anything heavier, reach for these words first.

7Negative prompts and the cargo cult

Negatives are genuinely useful — for what they're actually for. Back in Chapter 2 you saw the mechanism: a negative prompt points the update away from something. So use it for a specific recurring artifact ("extra fingers", "watermark") or a style you keep drifting into but don't want. Narrow, targeted, tested.

What you'll see pasted everywhere instead is the 200-word incantation — "mutated, deformed, bad anatomy, poorly drawn hands" repeated forty times. Mostly that just burns the window's attention on things your model was never going to draw anyway, and eats the same token budget you're fighting for on the positive side. A short, honest negative beats a long inherited one.

Same skepticism goes for quality tags. "masterpiece, 8k, trending on artstation" did real work on SD 1.5-era fine-tunes — but only because those specific models were trained on captions containing those exact phrases. They were learned magic words, not universal ones. On SDXL-class models they're mostly decoration. The lesson isn't "quality tags are fake"; it's test, don't inherit — which needs a method.

⚠️
Every token in a bloated negative is a token stolen from the ~75 you actually have. A wall of "bad anatomy ×40" isn't insurance — it's a tax you're paying to a superstition. If you can't point to the artifact it fixes, delete it.

8The prompt matrix

You already own the method — it's Chapter 2's X/Y grid, pointed at words instead of knobs. Fix the seed, fix the settings, and swap exactly one phrase per render. With vs. without a tag. Synonym A vs. synonym B. The subject at the front vs. buried. Because everything else is locked, any difference you see is caused by the one thing you changed — this is the fixed-seed A/B from Chapter 2, applied to language.

Do that a few times and you stop guessing. You build a personal phrasebook: the words that provably move your images, on your model. The community's giant tag lists are hypotheses — often written for a different model than yours. Your matrix is data. Trust the data.

A prompt is data, not prose

Here's the whole chapter as types. Model the prompt as a record of typed parts, render it in a fixed order, and the two rules that matter — subject first and mind the budget — stop being advice and become properties of the structure. The prompt matrix is just a comprehension over one field. Flip between the languages; the shape is identical.