# SUBTEX module – expanding simple TeX commands

Expanding simple TeX commands

Most of the (La)TeX commands found in BibTeX files are there to produce accented characters of one type or another. When Beastie parses such a file, it expands an assortment of such commands into the corresponding Unicode characters.

For example, the field

    author = {Ku\v rt G\"{o}del and Ren{\'e}e R{\o}mer}

will be processed as if it were

    author = {Kuřt Gödel and Renée Rømer}

Note the special case of `{\'e}` and `{\o}` – if this expansion process results in a single character appearing in braces, then the braces are discarded. This corresponds to a related, but distinct, special case in `bibtex`’s parsing of ‘accented characters’: `bibtex` treats such sequences, at brace-level one, as being a single character for the purposes of counting characters and generating labels. In Beastie, we support this case at any brace level, and go further by replacing the input sequences with the corresponding character.

Beastie recognises a fairly extensive set of such commands, and in particular should be able to manage all of the characters in ISO 8859-1 and the Unicode Latin-1 range, and most of the letters in the Unicode Latin Extended-A range, with a few exceptions where the LaTeX command is hard to find, which are therefore unlikely to be used in practice.

The set of supported commands is currently fixed, but may become expandable in future versions of Beastie.

Beastie also natively recognises Unicode characters in `.bib` files, so one could use either of the examples above in such a file, in this or any other script covered by Unicode.

Other commands in `.bib` files, such as `\emph{foo}` or `\LaTeX`, are not recognised by default, but you can add support for these within the Scheme layers (not within `.bst`) using the `user-char-command` function.

Beastie manipulates these internally using a ‘bstring’ type (for ‘braced string’). You will typically not manipulate bstrings directly, except if you use the combination of `ustring-iterator/bstrings` and `list->bstring` to iterate through a ustring in a bstring-aware way. Functions such as `uppercase-string/bst` from the `'bst` module use this to uppercase strings while leaving brace-level-1 content alone, in the way that `.bst` files require.

You may have a use for function `ustring->ustring`, which converts a ustring to a ustring in a way which allows you to control how braces and no-break spaces, or tildes in TeX terms, are managed. This is logically part of this module because it does this via a bstring. The function `untexify-string-or-list` wraps this in a way which might be convenient if you are generating `xexpr` structures, on the way to generating (X)HTML.

## Functions

Index:

- bstring->ustring
- bstring?
- list->bstring
- parse-subtex
- untexify-string-or-list
- user-char-command
- ustring->ustring
- ustring-iterator/bstrings

### bstring->ustring

`(bstring->ustring ts [:write write/display/#t/#f/] [:nbsp string])` : format a bstring?.

If the `:write` argument is `'display` or `#f` (the default), then produce a string for display; otherwise it should be `'write` or `#t` to produce a string that should be reparseable by `string->bstring`.

If, instead of `'display`, the selector is `'display/braces` or `'display/without-braces`, then the output ustring is or is not displayed surrounded by braces, respectively. If it is just `:display` then the outermost braces are skipped, but inner ones are included. This may seem a slightly fussy distinction, but recall that a .bib-file value like `"foo {bar}"` would be parsed as the bstring `#{foo {bar}}`, but the most natural way of printing this should end up with ‘foo {bar}’. I may add a settable global default (FIXME: cf subtex-load-hook code above).

The argument can alternatively be a list of `bstring-content?` objects.

Note that this default, preferring display over write since that is more common, is opposite to the internal `object->string` function.

### bstring?

`(bstring? x)` : return #t if the argument is a bstring.

A `bstring?` is a 'TeX string', returned by `string->bstring`. The bstring can contain objects of type `bstring-content?`.

### list->bstring

`(list->bstring l)` : given a list of `ustring?`, `integer?`, `bstring?`, or `'nbsp`, create a new `bstring?` object.

### parse-subtex

`(parse-subtex str [:on-error #f])` : parse a ‘subtex’ string or ustring, `str`, producing a `ustring?` object as result.

Given:

    One \emph{sharp} ep{\'e}e {here}

The function recognises both the TeX-style command `\emph`, and the TeX-style braces `{...}`, and and the ‘accent’ `{\'e}`, which it expands to ‘é’. The result is an opaque `ustring?` object.

The input string should be a ustring, a valid UTF-8 string of characters, or a list of integer codepoints.

On error:

- if ON-ERROR is `#f` (the default), then raise a beastie-error with subtag `'subtex`;
- if ON-ERROR is `#t` return STR;
- otherwise, return the keyword's argument as the function's value.

That said, the procedure as currently implemented doesn't throw any errors on parse failures -- that is, we aim not to object to any input. The only errors thrown are for internal coding errors, which shouldn't be caught here. I may remove this error-handling in future.

Although this function is exposed for use in end-user programs, it is not much needed, as the expansion in question has generally been done when data is read from a `.bib` file.

The type is called a 'TeX'-string because it primarily expands TeX ‘accents’ such as `\'e`, but it expands a few other (La)TeX control sequences as well. The list is unspecified, and not (currently) modifiable, but is intended to cover the control sequences often occurring in BibTeX databases.

In the special case of braces (any level) enclosing a single accented character, for example `n{\'e}e`, the character is expanded and the braces discarded, resulting in the bstring ‘née’; this tidies a common case often found in `.bib` files, and illustrated in the BibTeX documentation. This special case does not extend to ASCII-range characters (ie, `n{e}e` is parsed as ‘n{e}e’), but it does extend to single Unicode characters written directly (ie, `n{é}e` is parsed as ‘née’); this is slightly inconsistent, and the behaviour here may change if it becomes clear that an alternative can be clearly defined. The special case is strictly restricted to single characters, thus `n{\'ee}` becomes ‘n{ée}’.

A tilde in the input (`~`) is interpreted as a non-breaking space, in the usual TeX way. An actual no-break space character (Unicode U+00A0) is equivalent.

### untexify-string-or-list

`(untexify-string-or-list x)` : untexify a string or list. This removes braces and `"~"` from an input string, or a list containing strings. This wraps the function `ustring->ustring`.

### user-char-command

`(user-char-command key value)` : add a mapping for TeX-style `\\cmd` sequences. The `key` must be a `string?` or `ustring?`. The value may be a `string?`, `ustring?` or `integer?` codepoint, or a procedure of zero, one or two (currently no more) arguments, which evaluates to one of those types.

Thus, after

     (user-char-command "pounds" #"£")
     (user-char-command "poundstring" "lb")
     (user-char-command "poundtstring" #{lb})
     (user-char-command "poundnum" #xa3)
     (user-char-command "poundsproc" (λ () #"£££"))
     (user-char-command "poundsproci" (λ () "££"))
     (user-char-command "poundsprocii" (λ () #xa3))
     (user-char-command "emph" (λ (a) (make-ustring (sprintf "**~a**" a))))
     (user-char-command "concat" (λ (a b)
                                   (sprintf "(~a/~a)" a b)))

we might evaluate an input string as follows

    (ustring->string
     (ustring->ustring "ab\\pounds c \\emph{d} and \\concat e{f}"))
    -> "#"ab£c **d** and (e/f)""

In an input string, any unrecognised commands are turned into a string version of the command, without error. The only pre-defined commands are the single-character ones discussed in `parse-subtex`, below.

If you are interested in expanding commands within `.bib` files, then these commands must be defined before the `.bib` file is read.

### ustring->ustring

`(ustring->ustring str [:braces? #t/#f] [:nbsp "?"])` : rewrite a string? or ustring? into a ustring?, respecting ‘subtex’ conventions, and controlling the display.

This is very similar in effect to the combination of `parse-subtex` (qv) and `bstring->ustring`, but lets you control the resulting ustring somewhat.

If `:braces?` is `#t` (the default), then `{...}` braces in the input appear in the output. If it is `#f`, then they are suppressed (which is useful when the output result is not intended to be processed by TeX).

If `:nbsp str` is present, then that string is used when displaying a non-breaking space in the input. The default is `"~"` (ie, the usual TeX character), but setting this to a string containing a Unicode no-break space is an alternative. Thus, given the string `"a~é {\~n}{c}"` (with a no-break space in the middle), `(ustring->ustring s)` would evaluate to `#"a~é~ñ{c}"`, but `(ustring->ustring s :braces? #f :nbsp "+")` would evaluate to `#"a+é+ñc"`.

### ustring-iterator/bstrings

`(ustring-iterator/bstrings us)` : returns an iterator over the ustring, which will return single codepoint-integers on each call, just like the usual ustring iterator, except that it will return `{braced content}` as a single `bstring?` object, and non-breaking spaces (ie, from `"~"`) as a symbol `'nbsp`.

This, along with `list->bstring`, allows you to work through a ustring in a TeX/bst-compatible fashion, by preserving no-break spaces, and identifying all content outside brace-level 0. It means that, given a ustring `#"ab~c{d}"`, the operation

    (list->bstring (map values (ustring-iterator/bstrings #"ab~c{d}")))

is effectively a no-op.

Norman
2026 August 02