# Build and Generation Tools
{: .no_toc }

Builds the site and the PDF book, compares two builds, and regenerates the committed files that the build or a maintainer reads: the webfonts, the diagram font metrics, the packages' declared API, and the API snapshots the reference is compared with. The command-line conventions every tool shares are under [Command-line conventions](/Documentation/Development/Tools.md#command-line-conventions).

* TOC goes here
{:toc}

## tbdocs --- node builder/tbdocs.mjs
{: #tbdocs }

Entry point for the static site generator. Each caller adds its own flags to the bare `--src docs`, so take the invocation from the caller:

| Caller | Invocation |
|---|---|
| `build.bat` | `--src docs --check-audit-index` (plus anything passed through) |
| `checks.yml` (PR checks) | `--src docs --no-fetch-assets --check-audit-index` |
| `tbdocs-gh-pages.yml` (deploy) | the same, plus `--url` and `--baseurl` from the Pages environment |

Full invocation:

    node builder/tbdocs.mjs [--src <path>] [--dest <path>]
                            [--baseurl <prefix>] [--url <origin>]
                            [--dry-run]
                            [--no-offline] [--no-pdf] [--no-help-archive]
                            [--tolerate-missing-images]
                            [--fetch-assets] [--no-fetch-assets]
                            [--profile-offline]
                            [--check] [--no-check] [--check-audit-index]
                            [--check-findings <path>]
                            [--update-page-baseline] [--update-symbol-baseline]
                            [--symbol-gaps <path>]
                            [--serve] [--port <N>]
                            [--stall-timeout <seconds>]

A flag that takes a value accepts it as the next argument or as `--flag=value`; as the next argument, the value may not start with a dash.

| Flag | Effect |
|---|---|
| `--src <path>` | Source root. Default: `docs` relative to the working directory. |
| `--dest <path>` | Online-tree destination. Default: `<src>/_site`. The offline tree goes to `<dest>-offline`, the PDF tree to `<dest>-pdf`; with `--no-offline` or `--no-pdf` that folder is removed rather than written, so no tree from an earlier build is left beside the new one. Only a build into `docs/_site` updates the committed files a build compares with --- `builder/page-baseline.json`, `builder/symbol-baseline.json` and the help add-in's copy of the symbol index --- and writes the [help archive](#the-help-archive). A build into another destination checks against them and says what a build into `docs/_site` would change, so builds into folders of their own can run side by side without changing the working copy (see [where the drift guard compares but does not write](/Documentation/Development/Building.md#where-it-compares-but-does-not-write)). The build refuses a destination that is or contains `<src>`. Inside `<src>`, the destination must be, or be inside, a folder directly under it whose name starts with `_site`, `_serve` or `_pdf`; anywhere else there, the next build or `--serve`'s watcher would read the output back as source. |
| `--baseurl <prefix>` | Overrides `_config.yml`'s `baseurl`. CI uses it to set the GitHub Pages base path on fork deployments. |
| `--url <origin>` | Overrides `_config.yml`'s `url`; an absolute `http` or `https` URL. CI uses it so canonical URLs match the deployment origin. |
| `--dry-run` | Skip every filesystem write. |
| `--no-offline` | Skip the offline tree pass. |
| `--no-pdf` | Skip the PDF tree pass. |
| `--no-help-archive` | Do not write the [help archive](#the-help-archive). CI never passes it. |
| `--tolerate-missing-images` | Downgrade Phase 8's missing-image error to a warning, for a source tree that is mid-edit. |
| `--fetch-assets` / `--no-fetch-assets` | Force remote-asset vendoring on or off. Without either flag, a dev machine downloads missing YouTube thumbnails and GitHub user-attachment images, and a build with `$CI` set downloads nothing: a referenced but uncommitted asset is a hard error there. See [Authoring Pages](/Documentation/Development/Authoring.md#committing-downloaded-assets). |
| `--profile-offline` | Print per-substep timing for the offline tree pass. |
| `--check` | Run the link and site-integrity check over every tree the build produced. A failing check does not abort the build; it sets the exit code to 1, and its summary lines say whether links, integrity or both failed. |
| `--no-check` | Turn the check off again. Flags are read in order, so this wins over the `--check` in `build.bat`. |
| `--check-audit-index` | Implies `--check`, and also diffs the tree index the build derives from its own records against what was written to disk. A spurious entry would make the link check answer "exists" for a path that 404s in production. `build.bat` passes this. |
| `--check-findings <path>` | Implies `--check`, and writes the findings as JSON. Used by [`scripts/check_links_diff.mjs`](/Documentation/Development/Tools-Site-Checks.md#check-links-diff). |
| `--update-page-baseline` | Record this build's page and static-file counts in `builder/page-baseline.json` as the drift guard's new baseline. An ordinary build raises the baseline itself, so only a **fall** needs this flag. See [the page-count drift guard](/Documentation/Development/Building.md#the-page-count-drift-guard). |
| `--update-symbol-baseline` | Record this build's symbol-index URLs in `builder/symbol-baseline.json`. An ordinary build records new URLs, so only a URL the index has **stopped** publishing needs this flag --- and usually a pinned heading id instead. See [the symbol index and its drift guard](/Documentation/Development/Building.md#the-symbol-index). |
| `--symbol-gaps <path>` | Write the public symbols no page documents to a JSON file: each one's package, container, name, kind, and the page its container is on. Names in a package's `exclude_from_docs:` list are left out. |
| `--stall-timeout <seconds>` | How long the build waits with no task completing before it names the outstanding tasks and exits 2. Default: 120; fractions allowed. `0` disables the watchdog, and a wedged worker then hangs the build in silence. See [when a build stops instead of failing](/Documentation/Development/Building.md#when-a-build-stops). |
| `--serve` | Start the dev server (watch, rebuild, SSE live-reload). Each rebuild skips the offline and PDF passes. |
| `--port <N>` | HTTP port for `--serve`, a whole number from 1 to 65535. Default: 4000. |

A command-line error is reported on standard error before any work starts. These are errors:

- an unknown flag, or an unexpected argument;
- a flag without its value, or with an empty one (only `--baseurl` accepts an empty value, meaning the site root);
- a value the flag cannot use: a `--port` that is not a port number, a negative or non-numeric `--stall-timeout`, a `--url` that is not an absolute `http` or `https` URL;
- a `--dest` the build refuses.

Exit codes: **0** nothing to report (with `--serve`, the server was stopped with Ctrl+C); **1** the build or its check found a problem: a link or integrity failure, a failed build step, a fall in the page count, or a symbol-index URL lost; **2** a refused command line (a `--dest` the build refuses included), a build stopped by the stall watchdog, with `--serve` a failed first build or a port in use, or a crash.

### The help add-in's project file
{: #the-help-add-ins-project-file }

Every build packs the IDE help add-in's source folder, `add-in/`, into a project file and publishes it, so that [Help Add-In](/tB/IDE/AddIns/Help.md) can offer it as a download. `_config.yml` declares it:

```yaml
addin_project:
  src: ../add-in
  dest: tB/IDE/AddIns/downloads/tbDocsHelp.twinproj
```

`src` is relative to `docs/`; `dest` is the file's path in the online tree. The offline tree holds no copy. Its links to the file become the website's absolute URL and are not counted as unresolved. The build prints one line:

    add-in project: tB/IDE/AddIns/downloads/tbDocsHelp.twinproj, 21 files, 26.9 MB, with the offline archive, in 1 tree (143 ms)

The project holds:

- **Source files.** The files git tracks in `add-in/`, as they are in the working folder: an uncommitted edit is in, an unadded file is not. A second line names any untracked file left out. A `site.zip` in `Resources/HELP/` on the disk is never packed.
- **The archive.** The [help archive](#the-help-archive) this build wrote, so the add-in built from the file serves the pages without the internet. A build that wrote no archive packs none, and its line says `without the offline archive`; the add-in then shows the pages from the website.
- **The symbol index.** The `tB/symbols.json` this build wrote replaces `Resources/SYMBOLS/symbols.json`.

Line endings are normalised, so a Windows build and a Linux build write the same file. Packing needs Node and git, and neither twinBASIC nor Python.

`--dry-run` writes no file, and a source tree whose `_config.yml` declares no `addin_project` (the test fixtures) packs nothing. `--serve` writes the file into its one tree. A failed pack --- `git` missing, say --- fails the build with exit code 1, after the trees are written.

A `.twinproj` is not a publishable type. The [publish allowlist](/Documentation/Development/Tools-Toolchain-Tests.md#check-publish-policy) exempts this one path in the output trees and refuses a `.twinproj` anywhere else, or in `docs/`. [`test/addin-project.test.mjs`](/Documentation/Development/Tools-Toolchain-Tests.md#addin-project-test) tests what the file holds.

### The help archive
{: #the-help-archive }

Every build writes the zip the IDE help add-in serves the documentation from. The add-in embeds `add-in/Resources/HELP/site.zip` as a resource, and the build writes that file from the offline tree, before the build-time chart is added to `BuildInfo.html`. The step prints one line:

    help archive: add-in/Resources/HELP/site.zip, 2098 entries, 47.6 MB (0.7 s)

The archive leaves out what nobody looks up from the IDE: this Documentation Development section (`BuildInfo.html` and its chart, `assets/images/gantt.svg`, among it), the Videos pages and their poster frames, and the Challenges. It also leaves out every file readers download, which stays on the website: each is in a `downloads` folder beside the page that documents it. `HELP_ARCHIVE_SKIP` in `builder/help-archive-step.mjs` lists them, as patterns in the form `offline_exclude` takes. Every page still links to them, from the navigation, the search and the text; the add-in's server answers a request for a file the archive lacks with a redirect to the same file on the website. A file the Downloads page offers that the archive would hold fails the build, which names it.

The zip is too large to commit; `add-in/Resources/HELP/` is in `.gitignore`.

Only a build of the documentation tree into `docs/_site-offline` writes the archive. Builds of another source tree or `--dest`, `--serve`, `--dry-run` and builds without an offline tree skip it, and so does `--no-help-archive`. CI must not pass `--no-help-archive`, or the project file it publishes would hold no archive; `check_ci_workflows.mjs` fails when one does.

A file the build cannot write, or an archive that does not match the tree, fails the build with exit code 1. The build ends by checking the offline tree on disk against the archive again, now that the chart is in: the same files must be listed, less the ones the archive leaves out, and only `BuildInfo.html` and `assets/images/gantt.svg` may differ in size or CRC-32 from the archive's entry.

The build's lanes deflate the files, as tasks of its graph: the [Pipeline Stages](/Documentation/Development/Pipeline-Stages.md#helparchive-worker) page lists them. The offline tree is deflated while the search index is built, and the index's own file, `assets/js/search-index.js`, once it is written.

The reader on the twinBASIC side does no inflating of its own, so the zip format is fixed:

- One entry per file, no directory entries. A name is relative to the tree root, uses forward slashes and is UTF-8, with general-purpose flag bit 11 set. Entries are sorted by name in code-unit order.
- The DOS date and time are always 1980-01-01 00:00, and the version made by and needed is 20.
- No extra fields, comments or data descriptors: the CRC-32 and both sizes are in the local header and in the central directory.
- No zip64, so a tree of more than 65,535 files, or an archive of 4 GB, is refused.
- Files that are compressed already (`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.ico`, `.woff2`, `.woff`, `.mp4`, `.zip`, `.pdf`) are stored. Every other file is deflated, and stored instead when the deflated data is not smaller than the file. The level is 9 for a file deflated whole and 6 for one deflated in pieces, which is a file larger than 1 MB: the offline search index.
- A deflated file larger than 1 MB is deflated in pieces of 1 MB, which are joined into one deflate stream. Every piece but the last ends with a sync flush, and each piece is primed with the 32 KB of the file before it, so the stream is about the size of one made in a single pass. The bytes depend only on the piece size, not on how many pieces were deflated at a time.

Each file, or piece, is inflated again after it is deflated and compared with the bytes read. The build then reads the zip back, compares it with what it assembled, and checks every entry's headers against the list of files; it names any that differ. The writer is `lib/help-archive.mjs`.

## build_fonts.py
{: #build-fonts }

    python -m pip install "fonttools[woff]"
    python scripts/build_fonts.py

Regenerates the subset webfonts under `docs/assets/fonts/` from pinned upstream releases (SHA-256 verified), pinning the optical-size axis and keeping `wght` variable. The `.woff2` files are committed, so `build.bat` needs neither Python nor a network. The PDF pass aborts, naming this script, if a face is missing.

**Regenerating Inter means regenerating the diagram metrics too** --- see below. Changing a face, rather than refreshing one, reaches beyond this script; [Changing a typeface](/Documentation/Development/Builder.md#changing-a-typeface) lists every place the build names one. The script takes no options except `-h` and `--help`, which print the usage and start nothing.

Exit codes: **0** the faces were written; **1** a dependency is missing, an archive's SHA-256 differs from the pinned one, or a build step failed; **2** a refused command line.

## build_dot_metrics.mjs
{: #build-dot-metrics }

    node scripts/build_dot_metrics.mjs            # regenerate
    node scripts/build_dot_metrics.mjs --check    # fail if stale

Measures Inter's advance widths in a browser, from the committed `.woff2` files, and writes `builder/inter-metrics.json`, the table `builder/dot-metrics.mjs` installs into Graphviz before any layout runs. The JSON is committed and the build never runs the generator.

Run it after [`build_fonts.py`](#build-fonts) changes Inter. Forgetting shows up as [`check_dot_fit.mjs`](/Documentation/Development/Tools-Site-Checks.md#check-dot-fit) failing, with nothing naming the metrics. The script measures Inter by name, so giving the diagrams a different face means editing it; see [Changing a typeface](/Documentation/Development/Builder.md#changing-a-typeface).

Exit codes: **0** the table was written or is unchanged (with `--check`, it is current); **1** with `--check`, the table is stale; **2** a refused command line, a browser that would not start, or a crash.

## twin-index.mjs
{: #twin-index }

    node indexer/twin-index.mjs sync [--package <name> | --packages <a>,<b>] [--upgrade] [--ide <path> | --release]
    node indexer/twin-index.mjs sync --query [--builtin] [--package <name> | --packages <a>,<b>] [--ide <path> | --release]
    node indexer/twin-index.mjs analyze [--package <name> | --packages <a>,<b> | --all]
    node indexer/twin-index.mjs draft [--package <name> | --packages <a>,<b>]
    node indexer/twin-index.mjs derive [--check] [--out <file>] [--ide <path> | --release]

The package indexer, which finds the documentation work that a new twinBASIC build leaves. `sync` reads the sources of the packages and writes an `api.json` snapshot of each: every type the package declares and every member of each, with its visibility, kind and declaration. `analyze` compares the current snapshot of each documented package with the one the reference was last brought up to date with, and reports the public symbols to create a page for, to update and to remove. `draft` turns that report into the list of pages to write or change, each with its file, its parent and its permalink. `derive` writes `builder/package-api.json`, the package half of the [symbol index](/Documentation/Development/Building.md#the-symbol-index); see [The symbol index's packages](#package-api) below.

The build does not run it and it is not a gate. It sits in `indexer/` rather than `scripts/` because it keeps state of its own. It reads the sources with `scripts/lib/twin-api.mjs`, the repository's one scanner of twinBASIC declarations, which [`check_symbol_index.mjs`](/Documentation/Development/Tools-Toolchain-Tests.md#check-symbol-index) and [`check_twin_parsers.mjs`](/Documentation/Development/Tools-Toolchain-Tests.md#check-twin-parsers) test. [`test/indexer.test.mjs`](/Documentation/Development/Tools-Toolchain-Tests.md#indexer-test) tests it without a network, git or a twinBASIC install. The layout of `indexer/` and the format of `api.json` are in [`indexer/README.md`](https://github.com/twinbasic/documentation/blob/main/indexer/README.md).

**Where the packages come from.** The built-in and default packages (`VB`, `VBA`, `VBRUN` and the ones the IDE ships) are read from the files of a twinBASIC install, the unzipped IDE release. The newest build is the install `--ide` names, as its folder or its `twinBASIC.exe`; without it the tool takes `TB_IDE`, and then the newest `twinBASIC_IDE_BETA_<n>` folder on the Desktop. An older build is read from its `twinBASIC_IDE_BETA_<n>` folder on the Desktop. The BETA number comes from the install's `ide/build.js`, else from its folder name. `--release` downloads the same packages from GitHub releases instead, the latest and any older one by its tag, for a machine with no install; an install and the release zip of its build hold the same files, so either gives the same `api.json`. Nothing starts the IDE. The contributed packages are downloaded from TWINSERV, the package repository. A run that names only built-in or default packages does not contact it, so `sync --package VBA` works offline.

**Pinned versions.**{: #pins } A package the reference documents is pinned at the version its section's `indexed_from` names: a release tag such as `beta-x-1042` for a built-in or default package, a TWINSERV version such as `1.0.13.0` for a contributed one. `sync` reads each documented package at its pin, however new the newest version is, and lists the packages that have a newer version. `--upgrade` moves the packages named by `--package` or `--packages`, or every package without them, to the newest version instead. A package no section documents always follows the newest. So indexing everything again, after a change to the scanner, writes each snapshot from the version its pages describe, and no section moves to a new version until someone moves it. A package the store holds past its pin, after `--upgrade`, goes back to its pin at the next `sync` without `--upgrade`, and says so.

| Option | Command | Effect |
|---|---|---|
| `--package <name>` | `sync`, `analyze`, `draft` | Limits the command to one package, by the name the store has. |
| `--packages <a>,<b>` | `sync`, `analyze`, `draft` | The same for a comma-separated list. Not with `--package`. |
| `--all` | `analyze` | Covers every package in the store, not only the documented ones. Not with `--package` or `--packages`. |
| `--upgrade` | `sync` | Moves the packages named, or every package, to the newest version instead of keeping each at its [pin](#pins). |
| `--query` | `sync` | Lists what has changed upstream, and writes nothing; see [What `sync --query` lists](#sync-query). Not with `--upgrade`. |
| `--builtin` | `sync --query` | Compares the built-in and default packages too. |
| `--ide <path>` | `sync`, `derive` | `sync`: the twinBASIC install of the newest build. `derive`: the install to read, instead of that of the pinned build. Not with `--release`. |
| `--release` | `sync`, `derive` | Downloads the built-in and default packages from GitHub releases instead of reading an install. |
| `--check` | `derive` | Fails if `builder/package-api.json` is stale, instead of writing it. |
| `--out <file>` | `derive` | Writes somewhere other than `builder/package-api.json`. |

A name that matches no package is reported, the other packages are done, and the exit code is 1.

**What `sync` writes.** `sync` reads again only what changed.

| Path | Committed | Holds |
|---|---|---|
| `indexer/.packages/` | no | The store, a git repository of its own that `sync` creates. Each package has its `sources/`, its `Settings` file, its `api.json` and an `indexer.json` that records which version of the scanner wrote the `api.json`. A new store starts from the committed snapshots, so its history shows what changed in each package's API. |
| `indexer/.packages/manifests/` | no | What was last read: `built-in.json` (the newest build, its source, `install` or `release`, and its packages) and `contributed.json` (each TWINSERV package's publisher and the version the store holds). They never hold the path of an install, which contains a user name. |
| `indexer/.packages/downloads/` | no | Each package downloaded from TWINSERV, by name and version, so that moving a package to a new version and back downloads nothing twice. The store's history leaves it out. |
| `indexer/.packages/report.json` | no | The report of the last `analyze`. |
| `indexer/.packages/callouts.json` | no | The callouts labelled with a build older than the store's, from the last `analyze` of the whole reference. |
| `indexer/.packages/draft/` | no | What the last `draft` placed: a plan for each package, its pages in batches, and `manifest.json`, which lists them. |
| `indexer/snapshots/<group>/<package>/api.json` | yes | A copy of the `api.json` of each package the reference documents, of its pinned version, for `analyze` to compare with. `<group>` is `default`, `built-in` or `contributed`. |

`sync` judges a package by what the store holds, never by the manifest. A package is read again when the store lacks its sources or its `api.json`, or holds an `api.json` in an older format or of another version than the one wanted. A package whose `api.json` another version of the scanner wrote is indexed again from the sources already in the store, with no download. A clone has no store, so its first `sync` reads every package. A pinned version that TWINSERV no longer lists, and the store does not hold, cannot be read: `sync` reports it, and the exit code is 1.

**What `sync --query` lists.**{: #sync-query } With `--query`, `sync` reads nothing into the store and writes nothing at all: no sources, manifest, download or snapshot. It lists the contributed packages whose newest version on TWINSERV is not their baseline, with the date it was published: the baseline is the pin of a documented package, and the version the store holds of any other. It also lists the packages TWINSERV lists that the store does not hold, and those the store holds or the reference documents that TWINSERV no longer lists. The built-in and default packages are left out unless `--builtin` is given, or `--package` or `--packages` names one of them. They carry no version of their own, only the build's, so each package of the newest build is compared with its baseline build by content: its source files with those the store holds, and its public API, the symbols added, removed or changed as `analyze` counts them, with the API of the stored sources. A package with no stored sources of its baseline is compared by its API alone, with the committed snapshot. The newest build is the install `sync` would read, or the latest release with `--release`. A run that names only built-in or default packages does not contact TWINSERV.

**Which packages are documented.** A section of the reference documents a package when its folder under `docs/Reference/Default/`, `Built-In/` or `Contributed/` has an `index.md`, and the landing page's `package_name` lists the package (without that key, the folder's name is the package's). The page's `package_name`, `indexed_from`, `exclude_from_docs` and `exclude_kinds` are described under [Frontmatter and permalinks](/Documentation/Development/Authoring.md#frontmatter-and-permalinks); the indexer reads them and the build does not put them into any page. `sync` copies a package's `api.json` to `indexer/snapshots/` when the reference documents the package, the store holds it at its pin, indexed by the current scanner, and the two files differ. So a snapshot is only ever of the version the section's pages describe: a package moved past its pin by `--upgrade` keeps its snapshot until `indexed_from` names the new version, and a new section gets its snapshot from the next `sync`, even when the store was brought up to date before the section existed.

**What `analyze` reports.** By default it covers the documented packages and says how many others it left out. `--package` and `--packages` name any package. A package that no section documents has no pages to compare with, so every public symbol of it is listed as one to create, and so it is for every package under `--all`. A symbol is a type, or a name that a type declares: the Get, Let and Set of a property, and the overloads of a name, are one symbol, as the reference has one page for each. Only Public and Protected members of public types count, less what `exclude_from_docs` and `exclude_kinds` name. Each package is analyzed in one of two modes.

- **Update mode**, for a section whose landing page has `indexed_from`. The current `api.json` is compared with the snapshot committed at `HEAD`, provided that snapshot's `version` is the `indexed_from` tag. The report lists the symbols added (to create), changed (to update) and removed (to flag).
- **Audit mode**, for a section with no `indexed_from`, and when update mode has nothing to compare with: no committed snapshot, one in an older format, or one of another version, each with a warning. Every public symbol is matched by name with the `title:` of the pages in the section, and those with no page are to create.

**The callouts to check again.** A callout's label says which build of twinBASIC or which version of its package it is true of (see [What a callout applies to](/Documentation/Development/Authoring.md#what-a-callout-applies-to)). When the store holds a version of a package other than its section's `indexed_from`, the report lists each callout of the section labelled with another version, as a `recheck_callout` task: the build refuses the section's new `indexed_from` until each is checked and its label moved, or the callout removed. An `analyze` with no `--package`, `--packages` or `--all` also lists the callouts of the whole documentation labelled `BETA <n>` below the build the store's built-in packages were read from, and writes them to `indexer/.packages/callouts.json`. A `since BETA <n>` label is never listed, and neither is `timeless`: both stay true.

`analyze` exits 0 whenever it writes its report, whatever the report lists: the list is the product, not a failure.

**What `draft` places.** `draft` reads the last report and, for each package in it (or each one `--package` or `--packages` names), writes the plan `indexer/.packages/draft/<package>.json`. Each entry names the page file to write or change, the title of its parent, and its permalink, all taken from what most of the section's pages do: a new type goes in the folder the section keeps that kind of type in (an `Enumerations/` folder, or beside the classes), as a folder with an `index.md` or as one page as those types are; a type that another type returns, such as a collection's item, goes in that type's folder when the folder already holds a type; and a new member gets a page of its own or goes on its type's page, as that type's other members do. A permalink is filled in only when every page of the section has the permalink its folders give it, which is so for every section except VBA's; otherwise it is left to follow the sibling pages. Each entry also names a page of the same kind in the section to follow, and the source file that declares the symbol. A package with no section gets one in its group's folder, with a landing page titled and placed like those of the group's other sections. A symbol no page can be found for, such as a member of a type that has no page, is listed as unplaced. A symbol the build no longer declares is listed with the page that documents it. The pages are also split into batches, one file each, for whoever writes them in parallel.

**Moving the reference to a new build**, or one package to a new version:

1. `sync --query --builtin` lists what has changed. Run `sync --upgrade` with the new install, or `sync --upgrade --package <name>` for one package. The store moves to the newest versions; the snapshots stay at the pins, and `analyze` compares with the ones at `HEAD`.
2. Run `analyze`, then `draft`, and write the pages the plans list, where they say. Check each callout `analyze` lists as labelled with an older build or version on the new one, and move its label or remove it.
3. Set `indexed_from` on each section to the new build's tag (`beta-x-1005` is BETA 1005) or the package's new version. Run `sync` again, which writes the new snapshots now that the store holds each package at its new pin, and, for a new build, `derive`, which reads the newly pinned build. Commit the pages, the new `indexed_from`, the new snapshots and `builder/package-api.json` together. A snapshot committed on its own, or an `indexed_from` that is not the snapshot's `version`, sends the next `analyze` into audit mode.

**After a change to the scanner or the model.** The code that writes `api.json` is `scripts/lib/twin-api.mjs`, `scripts/lib/twin-declarations.mjs`, `indexer/lib/api-model.mjs` and `indexer/lib/vb6-preprocess.mjs`. A change to any of them, the `format` number of `api.json` included, can change what the same sources index to. The next `sync` indexes every package again, each documented one at its [pin](#pins), and writes again each committed snapshot that changes; commit them in the same commit as the change, with what `derive` writes. Otherwise `analyze` compares the output of the new code with that of the old and reports the difference as changes to the API. Run it without `--upgrade`, which would move the packages to their newest versions as well.

**The symbol index's packages.**{: #package-api } `derive` writes `builder/package-api.json`: every type the built-in and default packages declare, public or not, and the public members of each with their kinds, in declaration order. The [symbol index](/Documentation/Development/Building.md#the-symbol-index) takes its entries from the pages, and this file adds the kind of a member documented on a page of its own, an enumeration's values, and the interface a CoClass's members are declared on. It also says which public symbols no page documents. `derive` reads every package of the build the default and built-in sections are pinned at, from its `twinBASIC_IDE_BETA_<n>` folder on the Desktop, else from that build's release, which `--release` always downloads; `--ide` names another install. Sections pinned at more than one build make it refuse, unless `--ide` names one. It parses the packages as `sync` does, so it needs no store and no `sync` first. The snapshots cannot serve instead: they hold only the documented packages, and keep no declaration order. Packages are keyed by the project name, which is not always the package's: TwinBasicAssertions is `Assert`, and the three CEF builds are one `cefPackage`, whose APIs `derive` checks are identical. The file is committed and the build never writes it, because that needs the packages. Run `derive` when the reference moves to a new build, once `indexed_from` names it, and commit the file with the pages and the snapshots. `--check` compares instead of writing.

Exit codes: **0** the command finished: `sync` wrote every package it was asked for, or found it up to date; `analyze` wrote its report; `draft` placed its pages; `derive` wrote the file, or with `--check` found it up to date. **1** a package failed to sync (a pinned version that can no longer be had included), or one named by `--package` or `--packages` does not exist, or is not in the report (`draft`), and the others were done; with `derive --check`, the file is stale. **2** a refused command line, no twinBASIC install found, no snapshots to analyze (run `sync` first), no report to place pages from (run `analyze` first), packages that declare different APIs under one project name (`derive`), sections pinned at more than one build (`derive` without `--ide`), or a crash.

## convert_em_dash_separators.mjs
{: #convert-em-dash-separators }

    node scripts/convert_em_dash_separators.mjs            # rewrite in place
    node scripts/convert_em_dash_separators.mjs --check    # report, change nothing

Converts literal en-dash and em-dash characters in markdown source under `docs/` to the ASCII forms (`--` for en-dash, `---` for em-dash) that markdown-it's typographer converts at build time. The site forbids literal `–` and `—` in source, and this is the fixer if any slip back in.

It skips fences, indented code blocks, HTML blocks and inline code spans, and preserves each file's line endings. Its probes run in [`check_code_regions.mjs`](/Documentation/Development/Tools-Toolchain-Tests.md#check-code-regions). `--check` reports what it would change without writing, so it can serve as a gate.

Exit codes: **0** the dashes were converted (with `--check`, there were none); **1** with `--check`, a file holds a literal dash; **2** a refused command line, or a crash.

## survey_tooling.mjs
{: #survey-tooling }

    node scripts/survey_tooling.mjs                  # the summary, then every listing
    node scripts/survey_tooling.mjs --summary        # the summary only
    node scripts/survey_tooling.mjs --root <dir>     # measure another checkout

Measures the repository's own tooling for repetition and structure:

- code duplicated between files (matched token by token, so renamed copies still match);
- top-level functions defined under one name in several files;
- how the command-line tools read their arguments;
- packages imported without being declared in `package.json`;
- the import graph: imports that cross directories, files nothing imports, and the most imported modules.

It is not a gate, and nothing runs it: take a measurement before and after a refactoring. It reads only the files git tracks. `--root` measures another checkout, such as a worktree at an older commit that lacks the script. `perf/` is counted separately in the summary and left out of the listings unless `--include-perf` is given.

Exit codes: **0** the survey was printed, **2** a refused command line, a folder that is not a git checkout, or a crash.

## compare_trees.mjs
{: #compare-trees }

    node scripts/compare_trees.mjs                      # HEAD against the working tree
    node scripts/compare_trees.mjs --before <ref>       # any commit against the working tree
    node scripts/compare_trees.mjs --keep               # leave both trees and both build logs
    node scripts/compare_trees.mjs -- --baseurl /docs   # extra tbdocs arguments, for both builds

Builds the site twice and compares the online, offline and PDF trees byte for byte. One build is at a commit (`HEAD` unless `--before` names another), the other from the working tree as a commit would hold it, untracked files included. Use it on a change to `builder/` that should leave the output alone, or change it only where intended.

Both builds run from git worktrees under `.compare-trees/` (gitignored), without touching the index or the working tree, with `tbdocs --no-fetch-assets` and `CI=1`, so the committed baselines are never written.

Three regions differ between any two builds and are replaced before the comparison: the build's timings in `assets/images/gantt.svg`, the same chart inlined into the [Build Info](/Documentation/Development/BuildInfo.md) page, and the PDF title page's build line. Everything else must match.

A run takes about ten seconds. It is not a gate, and nothing runs it. A failed run leaves `.compare-trees/` for inspection; the next run removes it.

Exit codes: **0** the trees match; **1** the trees differ; **2** a refused command line, a git command or a build that failed to produce its tree, or a crash.

## build_speed.mjs
{: #build-speed }

    node scripts/build_speed.mjs bench [--tree <dir>]... [--runs <n>] [--cpus all,4] [--task a,b] [--out <dir>]
    node scripts/build_speed.mjs serve [--tree <dir>]... [--builds <n>] [--task a,b] [--out <dir>]
    node scripts/build_speed.mjs times <gantt.svg>... [--task a,b]
    node scripts/build_speed.mjs png <gantt.svg>...
    node scripts/build_speed.mjs profile <file.cpuprofile> --fn <name,name> [--depth <n>]
    node scripts/build_speed.mjs part <name> [--tree <dir>] [--file <file>]

Measures the build, for work on its speed: how long it takes, where in it the time is, and what one part of it costs alone. The times come from each build's [Gantt chart](/Documentation/Development/BuildInfo.md), since the build writes no timings file.

| Command | What it does |
|---|---|
| `bench` | Builds each tree as `build.bat` does, `--runs` times (default 3), on every CPU and on the first four, which is the nearest thing a development machine has to a CI runner. `--cpus` gives other counts, as `all` or a number. |
| `serve` | Builds each tree `--builds` times (default 5) on one pool of lanes, as `tbdocs --serve` rebuilds, without the offline and PDF trees. The first build is cold; the summary is of the rebuilds. |
| `times` | Prints the times in charts kept from earlier runs. |
| `png` | Renders charts to PNG files beside them, with puppeteer. |
| `profile` | Prints the inclusive time of each function `--fn` names in a V8 CPU profile, with what it calls to `--depth` levels (default 3). A function that calls itself is counted once. |
| `part` | Times one part of the build alone on one thread. |

`bench` and `serve` print one line for each build and then the range of each figure over the runs: when the first page is rendered, when the render ends and when the build ends. `--task` names main-thread tasks whose times are reported as well. One build's total says little on a machine that is doing other work, so compare ranges, and look at the charts. Each build's chart, and each `bench` build's log, is kept under `--out` (default `.build-speed/` in the checkout, gitignored).

A tree is a checkout: the one the tool is in, unless `--tree` names another. Repeat `--tree` to compare two, such as a branch's worktree and the main checkout. The trees are built in turn within each run, so that other work on the machine affects both. `bench` builds a tree in place, as `build.bat` does; `serve` builds into a folder of its own in the tree and removes it.

Take a profile for `profile` with Node's own option; the main thread's file is the one whose name ends `.0.001.cpuprofile`:

    node --cpu-prof --cpu-prof-interval 100 --cpu-prof-dir <dir> builder/tbdocs.mjs --src docs

The parts `part` knows:

| Part | What it times |
|---|---|
| `markdown-init` | The pieces of the `markdownInit` task: the link tables, the counts, the parser and the scan for placeholders. |
| `nav` | `computeNav`, each call on a fresh copy of the pages. |
| `index-slices` | The search index's add step against the slice count: its CPU in all and its longest slice. |
| `index-add` | The add step split into making the entries, tokenizing and stemming, lunr's `Builder.add`, and the postings written as JSON. |
| `index-terms` | What follows the adds: the merge, the inverted index's JSON, the vectors, the file's assembly and the offline copy. |
| `deflate` | A file deflated as one stream and as pieces, by level, piece size and priming. `--file` names the file; the default is the offline search index. |
| `archive` | The help archive's deflating on one thread, whole files and pieces apart, and the recheck of the archive. |

A part runs the modules of the tree it is given, so one checkout's tool measures another's code. The index parts, `deflate` without `--file`, and `archive` read files a build wrote: build the tree first. A part's time alone is not its time in a build, where it shares the machine with every lane.

It is not a gate, and nothing runs it. Its readers of the chart and of a profile are tested by [`build-speed.test.mjs`](/Documentation/Development/Tools-Toolchain-Tests.md#build-speed-test).

Exit codes: **0** done; **1** a build failed; **2** a refused command line, or a crash.

## render-book.mjs
{: #bookrender-bookmjs }

    node book/render-book.mjs <input.html> -o <output.pdf> [options]

The PDF renderer that `book.bat` calls. It converts the pre-built `_site-pdf/book.html`, which already holds all chapter structure, heading levels and outline entries; it does not read `_data/book.yml`.

It uses `puppeteer`, `paged.js` and `pdf-lib` directly, and sets `pdf-lib`'s `parseSpeed` to avoid ~32 seconds of needless event-loop yielding in a 100-second build; see [perf/README.md](https://github.com/twinbasic/documentation/blob/main/perf/README.md).

Key options used by `book.bat`:

| Flag | Effect |
|---|---|
| `-o <output.pdf>` | Output PDF path. |
| `--outline-tags h1,h2,h3,h4` | Heading levels to include in the PDF outline / bookmarks. |
| `--additional-script <path>` | Path to a script injected before paged.js runs. `book.bat` passes `perf\detach-pages.js`, which hides each finalised page from Chromium's layout tree and restores them all before `page.pdf()` runs. It cuts render time from ~104s to ~51s on a 1,638-page book. |

Exit codes: **0** the PDF was written; **2** a refused command line, an input or script that does not exist, a render that failed, or a crash. There is no 1.
