# UNICODE module – basic Unicode support

Basic Unicode support

Beastie does not have complicated Unicode-related requirements, but it does need to be able to read and write Unicode files, and to do language-sensitive sorting and case-switching.

### Character classes

The character class functions, in the `utils` module, are partially Unicode aware, in the sense that they will report the classes of characters in the BMP. These functions are `char-alnum?`, `char-alpha?`, `char-upper?`, `char-lower?`, `char-other-letter?`, `char-digit?`, `char-space?`, `char-punct?`, `char-cntrl?`, `char-symbol?` and `char-mark?`. There are also case-changing functions `uchar-upcase`, `uchar-downcase`, `uchar-titlecase`, and classification functions such as `uchar-alphabetic?`. See the documentation in the utils module for details of their behaviour, and see the Unicode Character Database (and its file collection) for detailed discussion of the properties in question.

## Ustrings

Various functions within beastie produce and handle ‘ustrings’, which should generally be equivalent to normal strings in use, but which have various other unicode-related operations defined. These principally consist of the unicode-related functions described just below.

Construct an empty ustring with `(make-ustring)`, or add content with `(make-ustring codepoint/string/ustring ...)`. The various `ustring-*` functions below are intended to broadly match the corresponding ordinary string ones. Since `ustring-append` creates a new ustring, that is an alternative way of creating a ustring from scratch.

Ustrings have write-form `#"abç"`, and the reader will parse this back into a ustring. Within the quotes, the only escapes recognised are `\\`, `\"`, and `\n`, with the last denoting a newline. Any other escaped characters are ignored, with a warning.

A ustring is an ‘applicable object’ in s7 terms: if `us` is a ustring, then `(us 2)` is the same as `(ustring-ref us 2)`, evaluating to the codepoint of `us` at index 2. Similarly `map` or `for-each` applied to a ustring will call the mapped function for each codepoint in the string.

## Reading and writing

A `unicode-reader?` is an object which turns a string or file into a sequence of Unicode codepoints. Associated functions are

- unicode-reader?
- make-unicode-reader/file
- make-unicode-reader/string
- unicode-reader-read
- unicode-reader-source
- unicode-reader-location

A `unicode-reader?` object is a `lexeme-source?` in the sense of the klipspringer module.

It may occasionally be useful to note that two `unicode-reader?` objects are deemed to be `equivalent?` if they refer to the same underlying object, string or file; they are `equal?` only if they are additionally at the same offset into the string or file.

Separately, we can encode lists of codepoints to UTF-8, and decode a UTF-8 string.

- unicode-decode/utf8
- unicode-encode/utf8
- unicode-encode1/utf8

## Unicode support in beastie

Beastie aims to be Unicode-aware by design rather than by accident. That is, input is parsed and handled as Unicode codepoints, rather than being bundled into UTF-8 strings and hoping for the best.

This means, for example, that the rules for allowed strings in `.bib` key and field names are expressed in terms of Unicode alphabetic characters (thus `नाम` is a valid key or field name, since it's a list of letters, but `a¶` isn't, since it includes a punctuation character). This obviously isn't a key beastie use-case, but it serves as a check on the ‘unicodeness’ of the code.

The conformance to Unicode is reasonably extensive, but not complete. There are two cases here.

If Beastie is built using the ICU library, then it has (some) locale support, and in particular it has language-sensitive sorting.

- unicode-get-locale
- unicode-get-locales
- unicode-set-locale!

If Beastie is build without the ICU library, either because it is not available or because it was suppressed at build time, then there is still Unicode support, but less sophisticated.

Further notes, applicable to the with- and without-ICU cases:

1. We properly support only the Basic Multilingual Plane (BMP) – characters outside of that, in the ‘astral planes’, are perfectly welcome in strings, but are not regarded as being included in any character class (so yes, you can happily have emojis in field values, but no, you may not have them in database keys).

2. There are various edge-cases, such as around title casing, that the code avoids (yes, the eszett!).

3. It's doubtless possible to confuse the string-classification algorithm by using one or other Unicode normal forms.

## Functions

Index:

- make-unicode-reader/file
- make-unicode-reader/string
- make-ustring
- symbol->ustring
- unicode-decode/utf8
- unicode-decode1/port/utf8
- unicode-encode/utf8
- unicode-encode1/utf8
- unicode-get-locale
- unicode-get-locales
- unicode-reader-location
- unicode-reader-read
- unicode-reader-source
- unicode-reader?
- unicode-set-locale!
- ustring->hash
- ustring->list
- ustring->string
- ustring->symbol
- ustring-append
- ustring-append!
- ustring-car
- ustring-index
- ustring-join
- ustring-length
- ustring-lowercase
- ustring-lowercase!
- ustring-ref
- ustring-split
- ustring-substring
- ustring-titlecase
- ustring-titlecase!
- ustring-tokenize
- ustring-uppercase
- ustring-uppercase!
- ustring<?
- ustring=?
- ustring?

### make-unicode-reader/file

`(make-unicode-reader/file fn ascii-characters?)` : make a unicode reader from a file `fn`.

If `ascii-characters?` is true, then the reader will return codepoints below 0x80 as characters rather than integers.

If the `fn` is `#f`, then this will read from stdin.

### make-unicode-reader/string

`(make-unicode-reader/string fn ascii-characters?)` : make a unicode reader from a string `str`.

If `ascii-characters?` is true, then the reader will return codepoints below 0x80 as characters rather than integers.

### make-ustring

`(make-ustring [codepoint/string/ustring ...])` : make a ustring, optionally adding content.

The list of arguments is the same as that for ustring-append, qv.

### symbol->ustring

`(symbol->ustring sym)` : returns the symbol `sym` converted to a new ustring

### unicode-decode/utf8

`(unicode-decode/utf8 s)` : decode the UTF8 string (an s7 string, regarded as a sequence of bytes encoded in UTF-8) into a ustring

### unicode-decode1/port/utf8

`(unicode-decode1/port/utf8 [p])` : decode a single unicode character from the input port, which should be UTF-8 encoded. If no argument is supplied, it defaults to `(current-input-port)`. This function isn't (currently) particularly robust against malformed UTF-8 input. It shouldn't collapse, but may skip characters when recovering.

Returns the next codepoint from the input, or `#<eof>`; returns Unicode replacement character on malformed input.

### unicode-encode/utf8

`(unicode-encode/utf8 l)` : given a list or iterator of Unicode codepoints, convert it to a UTF8 string (an s7 string, regarded as a sequence of bytes)

### unicode-encode1/utf8

`(unicode-encode1/utf8 i)` : given a single character or integer, convert it to a UTF8 string (an s7 string, regarded as a sequence of bytes)

### unicode-get-locale

`(unicode-get-locale [locale-name] [info])` : get information about the given locale.

With no arguments, returns an alist for the currently preferred locale. With one locale (string) argument, returns an alist of keys and values describing the locale. With two arguments, returns the (symbol) key information about the (string) locale, returning `#f` if the information is not available.

If the locale is given as `#f`, rather than a string, this is equivalent to the no-args case.

There is more information about Unicode locales in the ICU user guide.

### unicode-get-locales

`(unicode-get-locales)` : return a list of the available locales.

### unicode-reader-location

`(unicode-reader-location rdr)` : show the current location within the reader's source.

### unicode-reader-read

`(unicode-reader-read unicode-reader)` : return a single codepoint from the stream. Returns `#<eof>` at the end of the stream.

The unicode-reader object can also be called directly, as a no-argument function, to return the codepoint, and it can be called with argument `'location` to give the current location, as an alternative to `unicode-reader-location`.

### unicode-reader-source

`(unicode-reader-source rdr)` : show the source from which the reader draws its content, either as a filename, or as a "string", enclosed in quotes. The output is intended to be printable.

### unicode-reader?

`(unicode-reader? x)` : return `#t` if x is a unicode-reader

### unicode-set-locale!

`(unicode-set-locale! "locale")` : set the preferred locale for string comparisons. Passing a locale of `#f` resets this to the default. If the environment variable `$BEASTIE_LOCALE` is set, then that is the initially preferred value. Returns the name of the previously-set locale, or `#f` if the locale initialisation isn't aligned with a locale (for example if it is based on (default) rules).

### ustring->hash

`(ustring->hash u)` : return a hash integer for the ustring

### ustring->list

`(ustring->list us)` : convert a ustring to a list of codepoints

### ustring->string

`(ustring->string ustr [:display/:write/:readable/#t/#f])` : returns the ustring `str` converted to a (normal) UTF-8 string. If the second argument is `:display` or `#f`, then the string is suitable for display rather than for subsequent reading; otherwise, it is in a form which can be re-read to produce an equivalent object.

### ustring->symbol

`(ustring->symbol ustr)` : returns the ustring `str` converted to a symbol.

### ustring-append

`(ustring-append character|codepoint|string|ustring|list ...)` : append the arguments to the ustring. The procedure can take characters, codepoints, strings, ustrings, or a list of these. The result is a new ustring (cf ustring-append!).

### ustring-append!

`(ustring-append! ustring char|codepoint|... ...)` : append the arguments to the ustring. The replacement happens in place – the initial `ustring` argument is modified (cf ustring-append). The list of possible argument types is the same as for `ustring-append`.

### ustring-car

`(ustring-car us)` : return the first codepoint in the string, as an integer.

### ustring-index

`(ustring-index us cp [:start 0] [:end #f])` : Return the index of the first character in the ustring? `us` which is equal to `cp`, where `cp` is an integer codepoint or a character, or a (integer? -> boolean?) procedure [similar to `string-index`, but with a more restricted form for `cp`.].

The `:start` and `:end` keyword arguments delimit the scan, and default to the start and end of the string; these arguments are indexes into the string, with `end` indicating the index one past the last character to be considered; `end` may be `#f` to indicate the end of the string.

Returns `#f` if the character is not present.

### ustring-join

`(ustring-join ustrings sep)` : join a list of ustrings, by connecting them with the (ustring or string) separator.

### ustring-length

`(ustring-length us)` : return the number of characters (ie, codepoints) in the ustring

### ustring-lowercase

`(ustring-lowercase us)` : return a new ustring with the contents converted to lowercase

### ustring-lowercase!

`(ustring-lowercase! us)` : convert the ustring contents to lowercase, in place, and return the argument

### ustring-ref

`(ustring-ref ustring idx)` : return the codepoint (as an integer) at index `idx` (zero offset). Throws an `out-of-range` error if appropriate.

### ustring-split

`(ustring-split s c)` : Split an argument, which must produce an integer iterator, at a given character. The argument will most typically be a `ustring?`, but anything which iterates to produce integers, such as a list of integers, is acceptable. Returns a list of ustrings. Argument `c` must be a `string?`, `char?` or `procedure?`. See the procedure `string-split`, to which this should be closely analogous.

### ustring-substring

`(ustring-substring ustring start [end])` : return a copy of a substring of `ustring`, starting at index `start` and not including index `end`. Both `start` and `end` are zero-offset indexes of codepoints within the string. If `end` is absent or `#f`, then copy to the end of the string.

### ustring-titlecase

`(ustring-titlecase us)` : return a new ustring with the contents converted to titlecase

### ustring-titlecase!

`(ustring-titlecase! us)` : convert the ustring contents to titlecase, in place, and return the argument

### ustring-tokenize

`(ustring-tokenize us)` : split the ustring `us` into a list of substrings, where each substring is a maximal non-empty contiguous sequence of characters separated by whitespace.

The ‘whitespace’ here is anything which matches `char-wordbreak?`.

Returns a list of `ustring?`.

The argument is typically a `ustring?`, but can be anything which has an iterator which produces integers.

### ustring-uppercase

`(ustring-uppercase us)` : return a new ustring with the contents converted to uppercase

### ustring-uppercase!

`(ustring-uppercase! us)` : convert the ustring contents to uppercase, in place, and return the argument

### ustring<?

`(ustring<? u1 u2)` : return `#t` if both arguments are of type `ustring?`, and `u1` should be ordered before `u2`.

This is sensitive to the current locale/language.

### ustring=?

`(ustring=? s1 s2 [flags...])` : return true if the two arguments are ustrings or strings, and equal. Arguments of type `string?` are converted to `ustring?` before comparison.

There are no flags defined at present.

### ustring?

`(ustring? x)` : return `#t` if x is a ustring

Norman
2026 August 02