Verification: headless benches versus the real viewer

The rule

1. Anything the viewer interprets is proven only in the viewer

A mesh asset, a skin block, an .anim, a texture codestream, a particle system, a sound mix. Your own round-trips reproduce your intent and cannot catch a convention mismatch.

Learned: the whole animesh pipeline (skin encoder, .anim exporter, walk and graze behaviour) was verified at every step headless (rest-pose error 0.000004, .anim fidelity 5e-5, render strips, loop guards). The first real look showed the herd rendered as collapsed flat sheets. One look turned three "unknowns" into two bugs and one precisely diagnosed gap.
Learned: OpenSim's own mesh decoder accepted three format errors the viewer rejected (the UV domain key, the LLSD notation prefix, the zlib trailer); verify_mesh passing means nothing about the viewer.
Look after EVERY deploy that matters, not once at the end: one night of passing offline renders still left five faults only the viewer showed (invisible walls from a 101k-vertex face, wing cards shredded to specks, reeds through a jetty, a woodpile on a tall block, vegetables whitened in daylight), and an evening of passing sound checks left five only the ear caught (sound.md).

Build benches from the foreign system's own files. tools/render_inworld.py now does forward kinematics under the viewer's own skeleton file; before that it shared the encoder's parent table and passed a 1.56 m error at 0.00002. See animesh.md.

3. When the viewer disagrees, get a reference from the real tool

Upload one asset through the viewer's own uploader, pull it out of the server's bin/Asset.db (SQLite, opened read-only with ?mode=ro&immutable=1), decode it, and diff it against yours.

See: tools/decode_llmesh.py <asset-uuid> (header, skin block, domains), tools/cmp_mesh_struct.py (submesh keys), assets/firestorm-ref-skin.json (the decoded rigged reference).

4. The viewer's log and the server's asset traffic are fast oracles

The viewer's log (Firestorm.log) names mesh and texture failures ("Mesh header is invalid", "Error during mesh LOD processing"). Grep it after each change. On the server side, instrumenting the asset handler to log every asset request, miss and serve (with UUID, length and status) proved that the viewer had fetched a whole valid texture and still drew it grey, which pointed at decoding rather than delivery. That instrumentation is the opensim-patches/asset-instr patch: start the region with PARACOSM_ASSET_INSTR=1 and grep the log for GETASSET-INSTR.

Headless instruments (useful, but not the verdict)

tool	what it shows	blind to
`tools/render_scene.py [view] [time]`	the live scene (terrain, water plane at 20 m, time-of-day sky, objects from `list`) from named cameras, or from a visitor's own camera (`visitor`); `--season` previews a season	textures, particles, the viewer's shading; objects are simplified
`tools/preview_mesh.py`	a meshdata file's silhouette from several angles	textures, alpha, materials
`tools/raster_mesh.py`	a meshdata file with UVs, textures, tints and alpha-mask cutoffs, z-buffered	the 65,535-vertex face limit, texture decode rules, the viewer's lighting
`tools/render_inworld.py`	an animesh deformed the viewer's way, as side or front strips	textures unless given, the viewer's lighting
`tools/verify_anim.py`	an `.anim` decoded independently, compared with the pose that made it	how the viewer applies it
`tools/make_world_timelapse.py`, `make_world_orbit.py`, `make_world_montage.py`	the scene in motion (day, year, orbit) and a season x time grid	as `render_scene.py`

Motion checks: a still grid is the wrong medium for motion. Verify an animated GIF by the mean difference between consecutive frames, and do not trust a whole-frame metric when the thing that changes is a few percent of the pixels.

Reading state back from the world

inspect: a face's texture, materialID, colour, fullbright, shiny, and the live sound. A mask cutoff can be read through its material id (tools/mask_cutoffs.py).
list: position, size, rotation, colour, name, and the description (as shape). Scripts can publish their state into their description or name.
raycast: what the physics engine thinks is solid (physics-and-walkability.md).
avatars: where every avatar is, which way it faces, and where its camera looks.
The region database (bin/OpenSim.db): open read-only; to see the live WAL, open with mode=ro and WITHOUT immutable. The prims table shows GroupID and GroupMask; regionsettings the terrain textures. The LoopedSound column is stale; use inspect for sound.
The region log (bin/OpenSim.log): YEngine compile verdicts, console command output, watchdog warnings.

Looking in-world

The procedure that worked, described generically (any screenshot and input-automation tool will do):

Log in. Start the viewer (it remembers the login once the grid is added; operations.md), find the "Log In" button by OCR and click it. The viewer's UI is custom-drawn: UI Automation sees only the window frame, so use screenshots and OCR for anything inside it.
Wait for the avatar. Poll WorldBridge state or avatars until the avatar count goes from 0 to 1.
Place the camera. Teleport with a look direction: teleport with x,y,z and lx,ly,lz (or python tools/gaze.py <eye x y z> <target x y z>). The viewer applies the heading; its pitch settles near -0.25 whatever you ask. Do not reason about compass headings: read the measured camera lookAt from avatars and place the avatar back along it (python tools/tour.py <x> <y> [back] [up]), so the target lands mid-frame. (Both tools pass uuid= to teleport, which reads avatar; with one person logged in it makes no difference.)
Wait for the thing to stream. A new mesh can take several seconds; a re-uploaded texture has a new UUID. If a screenshot shows bare ground, check list and a raycast, wait, and look again.
Take a screenshot of the viewer window and look at it. For motion, take two about half a cycle apart.
Judge from the viewpoint that matters. A top-down view hid a 0.1 m sit gap that the sitter's side view showed at once. Right-click and Edit focuses the camera on an object. Esc drops back to the third-person camera. In mouselook, synthetic mouse moves do not turn the view (the viewer reads raw input).
Walk it. Hold movement keys to test stairs, ramps and doorways with the real avatar (an NPC's move-to-target can fly).
Close the viewer cleanly. Focus it, press Ctrl+Q, then click the dialog's "Quit" button (the LEFT one). OCR for "Quit" matches the right-hand "Do not Quit" first, which cancels. In mouselook the first Ctrl+Q may only leave mouselook: screenshot to confirm the dialog is up. Wait until the viewer process has exited.

Listening in-world

Record the host's audio output (a loopback capture) while the viewer plays the world, then analyse the capture (spectrogram, band-split RMS, matched filters). Transcriber labels on a capture are hints, not measurements. See sound.md for the analysis that worked.

When the viewer and a setting seem to disagree

Before blaming a viewer setting, find something of yours that DOES work in the same viewer. A one-frame texture animation that did not show was blamed on a viewer setting, while smooth-scrolling water was visibly flowing in that same viewer (the one-frame animation is simply not held; textures.md).