sprite_to_voxel_methodology.md

# Sprite → Voxel Methodology
## Translating Game Boy ¾-view overworld sprites into 3D voxel models

Worked example: Red's house exterior — `PALLET_TOWN` blocks 56/57/60/61, 64×48 px
(`reds_house_voxel.html`). The same pipeline applies to any GB building sprite.

---

## Core principle

A Game Boy overworld sprite is a fake-3D projection: it packs several different
3D facings into one flat image. Roof tops are drawn as if seen from above,
walls as if seen from the front, and sloped surfaces as diagonal silhouettes.
Voxelization is therefore **not** one operation — it is (1) classifying each
region of the sprite by which 3D surface it depicts, then (2) applying the
matching geometric operation per region.

Two invariants govern everything:

1. **Every visible voxel color comes from a sprite pixel** wherever the lookup
   lands inside the silhouette. Synthesis is allowed only for geometry the
   sprite implies but never paints (undersides, depth extensions, interiors),
   and synthesized cells must continue the sprite's own periodic patterns and
   4-shade palette.
2. **The sprite is ground truth, not the tile documentation.** The tile-grid
   doc described 1px ground flanks; the actual sprite has 5px insets on the
   ground floor and a 3-level wedding-cake silhouette. Always extract from the
   real pixels and verify any doc claims against the extracted mask.

---

## Stage 1 — Extraction

Sample the sprite down to its native resolution (uploads are usually integer
upscales; sample the center of each scale×scale block, never bilinear).
Quantize to the palette — for GB art expect exactly `#ffffff / #aaaaaa /
#555555 / #000000`. Anything else in the histogram means the sampling grid is
misaligned.

Compute the silhouette with a flood fill from the image border that spreads
**only through light pixels (luminance > 130)**. This threshold is load-bearing:
the black outline (#000) and the dark shading (#555) together form the
boundary. A naive "not black" threshold lets the fill eat the #555 shaded
flanks and corrupts the silhouette — that bug produced a 398-vs-148 pixel
discrepancy on the first pass here.

Output: a JSON of the palette plus one string per row (`.` = outside the
outline, digit = palette index). Dump the mask as ASCII and read it — this is
where the building's real structure becomes visible, and it is the input to
Stage 2. Tooling: `sprite_extract.py`.

## Stage 2 — Band classification

Segment the sprite rows into horizontal bands and assign each a facing. The
cues generalize:

| Cue in the mask / pixels | Reads as | 3D treatment |
| --- | --- | --- |
| Top band, plain fill with full-width course lines | Roof top seen from above | Lay horizontal (flat) |
| Silhouette tapering at a constant rate (here 2 px per row) with slat/stripe texture | Sloped roof surface; the taper rate **is** the slope | Stepped diagonal surface, 1 down per (taper rate) out |
| Band containing window/door frames | Vertical facade | Straight extrusion |
| Full-width band with a black underline sitting above an inset band | Ledge / awning overhang | Extrusion + protrusion |
| Dark `#555` runs beside a facade under a taper | Shadow on the wall beneath an eave | Leave as wall — the geometry above produces the shadow's meaning |
| Scattered light shapes on a dark field, bracketed by TWO full-width black rims, shallow band below the lower rim | The **inside of an open container** seen from above, with contents lying in it | Hollow tray: walls to the rims, floor slab, air between — never an extrusion |
| Ellipse drawn wider than tall (e.g. 9x5) | A horizontal circle seen from above — a mouth, a lid, a pot rim | Cut face of a round hull; the aspect ratio is the proof of the top view |
| Arcs above/below a round object's straight flanks, lowest point at the centre column, often a 1px #555 halo outside | The SAME circles seen curving — ground contact and mouth back-edge, i.e. depth, not narrowing | Strip them from the revolve; run the last body row's disc to the floor |
| A side band shearing sideways as it descends (¾-view) | The projection sliding a receding wall, not the wall's position | Un-project: the wall goes where the plan says |

The band table for Red's house, which Blue's house shares verbatim:

| Sprite rows | Content | Treatment |
| --- | --- | --- |
| 0–15, x16–47 | Coursed grey panel, black rims, highlight course | Flat horizontal top |
| 0–15, flanks | White/dark slats, 2 px-per-row taper | Sloped ends, 2:1 |
| 16–23 | Gable wall, 3 windows, slat flank pixels | Extrude; roof solid overwrites the corners |
| 24–31 | Awning slab, double black underline | Extrude + 2-voxel ledge front/back |
| 32–47 | Ground floor siding, door (own inner pane), 2 windows | Extrude; recess panes |

**Interpretation rules settled during this build — do not relearn them:**
a top-facing band must end up level everywhere (no synthesized skirts rising
through it); a tapering silhouette on a top band means *sloped surfaces in
elevation*, not chamfered corners in plan; and the slope's eave tips should
land where the sprite's taper pixels stop (here: just above the awning band).

## Stage 3 — Geometry construction

**Vertical bands** extrude straight back to depth `D` (26 here). Front and
back layers take the sprite pixel; interior layers take a **de-outlined**
color — if the pixel is black, walk inward up to 3 px for the first non-black
color. Without this, the side faces of the model are solid outline-black
slabs. With it, sides read as material with black front/back corner edges,
which is the correct GB-cartoon look. The back face becomes a mirror of the
front for free; that is sprite-pure and acceptable.

**Ledges** replicate the band's front pixels ±2 voxels in z past the walls.
The sprite's own black-underline rows become the visible dark underside.

**Recesses** delete the front voxel of every *non-black* pixel inside a
window/door rect. Frames stay proud; the identically-colored voxel behind
becomes the pane, one voxel deep. Rects with nested frames (the door's inner
window) produce layered relief automatically.

**The flat top** lays the top-facing rows horizontal. The band is shallower
than the house (16 rows vs 34+ of depth), so extend it by **cycling a mid-row
band whose period matches the course rhythm** (rows 5–8 here, period 4) —
this continues both course lines and slat columns seamlessly. Map the
outermost sprite rows to the front/back rims so the black-line/fascia trim
survives.

**Sloped ends** are driven by an elevation profile `T(x)`: flat at `YTOP`
over the plateau, dropping 1 voxel per (taper rate = 2) columns outward to
the eave tips. Build a solid of constant vertical thickness (4) following
`T(x)`, spanning the full roof depth including overhangs. Build order
matters: walls first, then the roof solid **overwrites** wall voxels it
intersects, then **trim** any wall voxel above `T(x)` so nothing pokes
through the surface. The wall strip left exposed beneath the slope shows the
sprite's own #555 shadow pixels — the sprite encodes this geometry.

**Outline pass**: cells on the roof's outer boundary get black (top layer),
grey (second layer), black (below) — reproducing the sprite's
black-grey-black fascia — and all interior undersides are dark.

Parameters used here, to tune per building: depth `D=26`; roof overhang 4 in
x and z beyond the walls; slab thickness 4; `YTOP=35`, derived so the eave
tips (after the 10-step drop) land one voxel above the awning band, matching
where the sprite's taper ends.

## Stage 4 — Color sourcing off the sprite

For any roof cell, map z to a sprite row (`roofSy`), then look up `(x, row)`.
If that lands inside the silhouette, use the pixel. If it lands outside
(overhang extensions), continue the sprite's periodic texture: the slat
rhythm is period 3 with a per-side phase (left flank white at `x % 3 == 1`,
right at `x % 3 == 2` — derive the phases from the actual pixels, and mind
negative-modulo semantics in JS). Plain top areas fall back to the mid grey.
Ground plane and path are presentation-only and the single place non-palette
colors are permitted.

## Stage 5 — Verification (non-negotiable)

Every bug in this build was caught by one of these, none by eyeballing alone:

1. **Dual implementation parity.** Build the identical algorithm in a
   reference implementation (Python) and in the shipping runtime (JS in the
   HTML). Diff total voxel count and post-cull shell count — they must match
   exactly (final build: 58,356 / 14,169).
2. **Numeric asserts on intent.** Flatness: the set of y-layers above the
   walls must be exactly the slab layers. Slope: the top-surface profile at
   mid-depth must read tips → 1-per-2 steps → flat plateau, mirrored. Zero
   wall voxels with `y > T(x)`. Full wall coverage by the roof footprint.
3. **Isometric preview.** A ~60-line painter's-algorithm render (sort by
   `(x+z, y)`, draw top/left/right faces of shell voxels) catches texture and
   layering mistakes cheaply before touching the runtime.
4. **Hidden-face culling.** Drop voxels whose 6 neighbors all exist before
   instancing; render the shell as one InstancedMesh.

Tooling: `voxel_build_verify.py` (builds, asserts, renders previews).

## Repeat checklist

1. Obtain the sprite; sample to native resolution via block centers.
2. Extract palette + silhouette (light-only flood fill, threshold 130);
   review the ASCII mask — rendered large, not hand-counted.
3. Name the real object first (including whether it is hollow, round or
   thin — see "Beyond the house"), then segment rows into bands using the
   Stage-2 cues; write the band table as prose, one line per row range
   with where each band lands, before writing any geometry code. The
   correct reading makes the row arithmetic land exactly.
4. Measure taper rates from the mask; derive `T(x)`, `YTOP`, overhangs, `D`.
5. Build: extrude verticals (de-outlined interiors) → ledges → recesses →
   flat top (mid-row cycling) → sloped solids (overwrite, then trim) →
   outline pass → ground presentation.
6. Verify: parity counts, profile/flatness/poke asserts, iso preview.
7. Ship: embed palette + row strings in the HTML; the builder runs
   client-side and doubles as the reference implementation of the algorithm.

## Applying this in the mod

Because tilesets are shared, the band table can be keyed by tile id rather
than by sprite: each id gets a treatment record (extrude / lay-flat / slope,
plus ledge, recess, warp flags) in the spirit of
`mods/DRAMATIC_SHAPE/data/voxel_heights.lua`. Pinning blocks 56/57/60/61 with
the profile above voxelizes Blue's house identically for free and propagates
to the Fuchsia City, Route 25, and Saffron City instances of the same art.
The door's lower-left tile (27) is the warp/walkable tile — keep its recessed
front face aligned with the collision cell so the 3D doorway matches
`REDS_HOUSE_1F`'s warp.

### What shipping it settled

The mod implements this as `mods/DRAMATIC_SHAPE/lib/Buildings.lua`, driven by
a `buildings` list in the profile. Three things changed from the sketch
above, each for a reason worth keeping:

**Key the band table by the building's tile GRID, not by tile id.** A tile
id is not a band: tile 23 is the house's awning course *and* the top of its
roof *and* the eave course that ends Oak's lab's much taller roof. Matching
the exact grid (`../buildings/` catalogues one per building, with every
map that places it) is unambiguous, still shares one entry across all seven
placements of Red's house, and cost nothing to verify — a scan of all 222
maps returns exactly the catalogued placements.

**Measure everything measurable.** Only the band table needs a human to read
the drawing. The silhouette, the taper rate, the eave height and every
window and doorway come off the pixels: a pane is a non-black region the
drawing seals behind its own black frame, and `T(x) = YTOP - topRow(x)`
falls straight out of the mask — which is also what makes `YTOP` stop being
hand-tuned. The eave tips landing one voxel above the awning, tuned by hand
here, then happens by itself.

**Depth is the plot, and the drawn row → depth mapping has a direction.**
In a diorama `D` is free; on a map it is the building's footprint (48px for
Red's house, 64 for the lab), which is why the roof band has to be cycled so
far. And the drawing looks at the roof from the north: its top rows are the
FAR edge and its bottom rows the eave over the facade. Getting that backwards
is invisible in the counts and in a symmetric preview — it shows up in game
as a fascia along the wrong rim.

One colour note: the outline pass's grey fascia band (`GREY` at `t-1`) is
right for the raw GB palette but comes out white once the atlas is
recoloured, turning every sloped end into a black-and-white zip. The
drawing's own eave is black / `#555` / black, and using that reads correctly
under every palette.

## Beyond the house: the forms later objects added

The house is all solid masses — every band either lies flat or extrudes.
Later objects forced the taxonomy open, and each addition came from the
same root move: **name the real 3D form first, then ask which surfaces the
drawing shows.** The recurring failure at every step was the *extruded
picture* — and it has a second-order form that survives re-segmentation.
The Bike Shop's toolbox was re-read from "a prop" into "a cabinet with a
pump beside it": named parts, correct plot, de-outlined sides, and still
wrong, because the region read as a cabinet *front* was the inside of an
open box seen from above. Naming the parts is not enough; every REGION
must answer "what surface of the real object is this?" The reliable
arbiter is arithmetic: the correct reading makes the drawn row counts land
exactly (the toolbox: 1 back-wall rim + 6 interior rows + 1 front rim = 8
= the one-tile plot depth). Forcing rows to fit means the reading is wrong.

**Hollow forms.** An open container is the one shape whose model must
contain AIR, which no band table or extrusion can produce. The tray
treatment builds four walls to the drawn rims, lays the top-view band on
the floor of the cavity (its contents — a wrench — come along free, since
they are just pixels of that band), and leaves the space between empty.
Two rules only containers hit: the pane-recess pass must never run on a
one-voxel wall (it deletes the front voxel to expose the one behind, and
there is nothing behind — the wall becomes a hole), and the hollowness
needs its own verification assert, because a later change that refills the
cavity leaves every count looking plausible.

**Round forms.** A drawn ellipse wider than tall is a horizontal circle
seen from above — that one aspect-ratio measurement settles the whole
reading. Straight flanks give diameter and height at once (round in plan,
so drawn width IS depth — the one depth never authored). The arcs above
and below the straight run are the same top and base circles seen curving:
ground contact and mouth edge, not narrowing — revolving them puts the
object on a stem. The hull's chord representation stores one z-interval
per column/row, so a taper is expressible (re-cut the chords, squeeze the
art into the narrowed span so the rim outline survives) but a hollow ring
needs a second chord. Voxel resolution bounds taste: on an 11-wide object
a one-step taper reads as damage and two steps as a cone; pick the step
count and derive the amount.

**Thin forms.** A line drawing cannot be thick. The air inside a bicycle's
frame is what makes it read as a bicycle; extrude each stroke 5 voxels and
the side faces of neighbouring strokes close every gap off-axis — six
bikes become one dark mass. Standee thickness is a vocabulary
(`PINNED_DEPTH`: 1 for paper, 2 for plates and side-on vehicles, 5 for
silhouettes, 10 for objects with a body), and when a standee looks wrong
the first move is to dump the detector's mask — if the mask is a clean
object, thickness is the problem, not segmentation.

**Authored masks.** When a drawing shares its tiles and shades with what
it is painted into, nothing automatic can separate them; the profile
carries a pixel mask instead. A person becomes a `figures` card (flat,
leaning with the camera, standing on its feet — because GB character art
is face-on iconography); an object becomes a `mounted` slab (fixed in the
world, holding the wall's plane, keeping its drawn elevation — because a
side-on drawing is a plane parallel to the wall). And when the backdrop is
a *regular* pattern, the mask should be MEASURED, not hand-drawn:
composite the plain backdrop tile over the same grid and flood from the
border through pixels that still match it — what the flood cannot reach is
the object, sprite-pure and exact.

**Verification, extended.** Isometric previews miss what only the game
shows: shoot in-game at both the ¾ rung and the low rung (front-face holes
and proportion errors are invisible from above), crop and NEAREST-upscale
before judging, and remember the flat rung renders no model at all. Two
cheap renders beat argument: the front-most voxel per (x, y) laid beside
the composited drawing catches anchoring and texel leaks instantly, and
the same render with sunk voxels flagged turns the recess pass into
something you look at. When shared builder code moves, a saved count
baseline diffed after every edit (mind the line endings) is what proves a
generalization is an identity for every model that already shipped.