# Beastie – an experimental processor for BibTeX and friends

- Usage
- Beastie is not the same as BibTeX
- Other overlapping resources

Beastie is an experiment in parsing and processing BibTeX files.

Home page: `https://purl.org/nxg/dist/beastie`

Beastie is:

- A parser for `.bib` files, amongst others.
- A framework for processing the parsed bibliographic data, written in a dialect of Scheme, extended with suitable parsing and processing primitives.
- A implementation of the `.bst` BibTeX Style Language, using that framework.

The goals of this project are:

- to explore ways of processing `.bib` files which aren't exclusively based on `.bst`;
- to provide a second implementation of the BST language; and
- to initiate some sort of conversation about BibTeX standards, and their documentation, broadly construed.

The first and second of those points aim to get away from BibTeX being a single-implementation system, where the only thing that can reliably process a `.bib` file is the `bibtex` program. This isn't quite the case at present, but combined with the third documentation point, it's quite hard to process BibTeX input in a confident way.

The `.bst` language counts as rather niche. It has substantial amusement value, but expertise could hardly be said to be widespread (of course, one could complain that Scheme has some niche aspects, but it also has its numerous compensating virtues).

The implementation of the `.bst` language is intended to be fairly complete, in the sense that it is nearly a drop-in replacement for the `bibtex` program (as described in the ‘Writing BibTeX styles’ document, ‘btxhak.pdf’, on the bibtex package page). Some of the differences from BibTeX will be bugs, but others are deliberate mild deviations, as discussed below.

Beastie also provides a framework for writing alternative bibliography tools. The ‘experimental’ status means that it is not yet fully-featured in that respect. The fact that it implements the `.bst` language confirms its scope, and acts as some indicator of ‘correctness’, and the `examples/` directory in the distribution illustrates how one would go about writing a from-scratch style file, but there isn't (yet) a drop-in Scheme-based alternative to, for example, `plain.bst`.

It is a goal of Beastie to go slightly beyond BibTeX, and to make it a little easier to use `.bib` files to generate HTML or other formats, or for the output of beastie to be ingested into other processing pipelines which are not based on (La)TeX. This has informed some of the choices made.

Another limitation of BibTeX, and its ecosystem, is that, although it provides fairly comprehensive support for STEM-style bibliographies, it starts to struggle with citations much outside this area. For humanities or legal bibliographies, the most active support currently comes from BibLaTeX, possibly because the intricacy of the implied support requires a more approachable language than `.bst`.

It's also a goal of this project to produce a v1.0 at some point.

### Features

- Programmatic manipulation of BibTeX bibliographies in a high-level language. That language is Scheme, with notes on the Beastie dialect. And using that...
- An implementation of the `.bst` BibTeX style language.
- `.bib`, `.bst`, `.aux`. and Markdown parser/converters (and `.json` because why not?).
- A few minor extensions to the `.bst` language.
- Slightly more informative error handling.
- Parsing of `.bib` files promptly expands a selection of `{\"u}`-style escapes into the corresponding characters (this makes it easier to use this tool for generating HTML and other non-LaTeX bibliographies).
- Unicode-aware throughout, in terms of case-folding and word-boundary recognition (with some qualifications).

### Limitations

- There are one or two edge-cases where the `btxhak` document doesn't make it fully clear what counts as ‘correct behaviour’. There are also some places where the implementation deliberately extends or mildly deviates from BibTeX behaviour – see below.

- The `.bib` parsing doesn't attempt to round-trip the file contents: inter-entry text is discarded when parsing, and `{\"u}` escapes are expanded promptly. This isn't so much a limitation, as a restriction of scope.

- The Markdown support is fairly basic – it's a side-project here. It implements most of the Gruber spec, but not all, and in particular it doesn't know anything about HTML content.

- Although the project's goal is to make it easier to write bibliography processors, the provision within Beastie is still at the level of adding parsing primitives, and not much at a higher level. That is, the distributed example scripts contain much that would probably become ‘library’ code in a mature state.

As far as the `.bst` language is concerned, it aims to support it quite fully, in the sense that it will produce the same results as BibTeX, for the standard styles. However it doesn't support it slavishly, and there are some deviations noted below. It also aims to slightly improve `.bst` error reporting, and adds a couple of extra `.bst` builtin functions, in both cases with the aim of being useful for debugging or developing `.bst` files (eg, `show.stack$$` and `printf$$`).

Some of these points are discussed more extensively below.

### Future plans (tentative)

Beastie is still a development project (though there is a firm determination to get to a version 1.0 before too long!). Some acknowledged gaps are:

- Beastie contains support for Unicode both using the Unicode consortium's ICU library, and also using a local library for parsing and serialising Unicode. The main limitation of the latter is that is has only a very naive collation algorithm. This is unlikely to change, however, since it's mostly regarded as a fallback. The support for locales has not been thoroughly tested; bug reports are welcome.

- The documentation is still somewhat ragged. There's quite a lot of it, but it's grown rather organically and could do with being rationalised.

Longer-term developments (that is, things which might wait until after a v1.0):

- An implicit goal of Beastie is that it provide the sort of foundation upon which one might experiment with implementing Citation Style Language. But that’s definitely a longer-term project. That's the sort of project which would use the ‘library code’ mentioned above.
- Develop a fuller replacement for (eg) `plain.bst` in Scheme, as a proof-of-concept?

### Acknowledgements

Most of this is implemented in an embedded copy of s7, from ccrma.stanford.edu. Thanks to Bill Schottstaedt for creating s7, and for advice on some of its intricacies.

Thanks to Karl Berry and Paulo Ney for comments, and for functionality suggestions.

## Usage

There is a manpage.

### Processing `.aux` files using `.bst`

If you call beastie with

    % beastie foo

then it will parse the aux file `foo.aux`, looking for `\citation`, `\bibdata` and `\bibstyle` commands, just like BibTeX, and process the bibliography with the indicated `.bst` program. That is, this is (still tentatively) intended to be a drop-in alternative to `bibtex`, though I say this with some diffidence, since it hasn't really been pushed around much. There are some deviations from the authoritative BibTeX behaviour, noted below.

### Processing using Scheme – programming

If you call beastie with

    % beastie foo.scm [args...]

then the program `foo.scm` is interpreted as Scheme, extended with the functions described below. My assertion is that this is a more enjoyable way of generating bibliographies than the standard `.bst` language.

That is documented elsewhere.

Any `args...` are available in the program in the global variable `*command-line*`; there is a `getopt` function which may be useful if you are writing a script.

If you invoke beastie with the `-r` option, or if you make `repl` and invoke the program `repl`, then you're dropped into a basic REPL.

While beastie still has a version number less than 1.0, everything here should be regarded as provisional.

### Converting BibTeX

If invoked with the name of a `.bib` file,

    % beastie foo.bib

then the given file will be parsed and re-written as a `.bib` file. If you supply the `-O` option, then you can select an output format; if this format is `bib`, `json`, or `sexp`, the output will be in the corresponding format.

You can do the same thing with `beastie -I bib <foo.bib`.

### Converting Markdown

The program also contains a basic Markdown parser. This was because it was sort-of easy to do, and convenient for documenting the program. Its ambitions are modest – it is intended to implement the core of the format, but not to be a comprehensive implementation.

You can experiment with the Markdown support with

    % beastie foo.md

The default output is in the form of ‘x-expressions’ – XML represented as scheme s-expressions, which is easy to wrangle (see the `xexpr-*` functions below). The output options are:

- `-O sxml` (default) : x-expressions.
- `-O xml` and `-O xhtml` : output the result as XML – in the former case, this is a single `(div ...)` expression; in the latter this is wrapped so that it forms a complete XHTML document (the distinction in the naming of these two outputs is somewhat arbitrary).
- `-O python` : something easily ingestible with Python (specifically listxml).

You can do the same thing with `beastie -I md <foo.md`.

The implementation covers a decent fraction of the Gruber spec; the only significant omission is that it currently knows nothing about inline HTML.

There is experimental built-in support for references (perhaps ‘illustrative’ support would be better), in the style of RMarkdown. Specifically, if a Markdown document contains references marked up like `[@foo]`, then the multiple values returned from `parse-markdown-file/metadata` will be a parse-tree and a collection of ‘metadata’. From that metadata, you can extract (with `(metadata/type 'citation)`) a list of citations and the place in the parse-tree where they appear. The latter are the `(cite "foo")` elements in the parse-tree, which can therefore be edited retrospectively. A way of using this is illustrated in `examples/bibliography-in-markdown.scm`.

There is an alternative illustration of how one might support citations in `examples/bibliography-in-markdown-alt.scm`; see the comments at the top of the file for details.

Similarly, `(metadata/type 'annotation)` will produce a list of `("key" "value")` lists containing annotations from the input Markdown file. The annotations are of the form

    ---
    key: value

(ie, something resembling a YAML hash; this is the only YAML which is recognised). It happens that

    Text
    ---
    key: value

will also be recognised as metadata, rather than a section heading. So don't write that.

## Beastie is not the same as BibTeX

Beastie is not intended to be a complete and exact replacement for BibTeX, and certainly not ‘bug-compatible’. It should be broadly compatible with BibTeX in ‘normal’ use (for most values of ‘normal’), but may treat edge cases differently.

These differences are discussed on the dislocations page.

## Other overlapping resources

### Beebe: comments on bibliography prettyprinting, and syntax

There is a discussion of BibTeX (and some lamentations) in

    @article{beebe93,
      author =       {Nelson Beebe},
      title =        {Bibliography Prettyprinting and Syntax Checking},
      journal =      {TUGboat},
      year =         {1993},
      volume =       {14},
      number =       {4},
      pages =        {395--419},
      month =        dec,
      url =          {https://tug.org/TUGboat/tb14-4/tb41beebe.pdf}
    }

I agree with most of the observations in Beebe's paper, and have not aimed to duplicate his collection of BibTeX formatting and validation bibliography tools. Although beastie can effectively do some normalisation of an input `.bib` file (with `beastie -O bib foo.bib`), this is intended to be a convenience, rather than a cleaning operation, as such, or a prettyprinting one. I would expect that `beastie` would have no problem parsing anything `bibclean` emits, and failing to do so would be regarded as a bug in `beastie`.

This article also contains useful (and otherwise very hard to find) information on the ‘Scribe’ format, which BibTeX effectively inherits. I have not attempted to support any part of the Scribe format which goes beyond what I have seen as ‘typical’ of BibTeX format files, and have felt free to be rather opinionated about what counts as ‘typical’ there. Beebe notes that both the lexical content, and the grammar, of `.bib` files are under-specified (to put it politely), and makes some suggestions.

- He suggests `[A-Za-z][-A-Za-zO-9:.+/']*` as a pattern for names (keys and field names). I have seen `_` and `&` in `.bib` files, and ADS routinely has keys starting with years, so these seem reasonable additions, to me. I'm willing to be slightly bolder than Beebe, and forbid the single-quote, `'`, as a name (it's just asking for trouble). I've therefore gone with `[0-9]*[A-Za-z][-A-Za-z0-9:.+/_&]*` as the name pattern in beastie.

- Beastie supports `%`-comments: characters from `%` to the end of line are regarded as whitespace (the document suggests a slightly more complicated rule, including gobbling leading whitespace on the folowing line, but I don't fully understand the suggestion there, and believe that what I have implemented here implies the rule there, since beastie doesn't aim to make its parse round-trippable).

- And beastie supports the suggested `@include` form.

There are a number of tests in the `bibclean` documentation, some of which have been imported, in adapted form, into the tests in `test/`. But a large fraction of these are testing the various normalisations that bibclean performs, which beastie doesn't attempt.

### Other software

There are a number of miscellaneous BibTeX utilities at CTAN. Some of these overlap with Beastie.

- `bibtex8` is a version of `bibtex` which is 8-bit clean and has larger internal capacities (and which is sometimes the actual program invoked as`bibtex`). This can use a range of national character sets.
- `bibtexu`, is a version of `bibtex8` which is Unicode-aware, in that it can do Unicode-informed case-changing and sorting.
- Bibtool overlaps somewhat with Beastie, in that it's a library for handing and processing `.bib` databases, and a binary which uses the library.
- `biblatex` relies on either BibTeX or `biber` to parse the bibliography, and uses TeX as the formatting language for bibliographies. Separately from the implementation details, BibLaTeX defines a substantially enlarged repertoire of `.bib` entry types and fields: such an expansion in expressivity is valuable, but few of them have been implemented in `.bst` style files.
- There is Perl support for BibTeX in Text-BibTeX, which includes a C library (`btparse`) for parsing `.bib` files, along with a discussion of their syntax.
- Finally, in Python, there are
  - bibtexparser, a `.bib` parser in Python;
  - biblib a `.bib` parser, which also includes a reference grammar for `.bib` files;
  - bibulous, which provides programmatic access to a parsed `.bib` file, and supports an attractively compact templating language for describing bibliograpy formats.

Norman
2026 August 02