`@bgotink/kdl`

This package contains a parser and stringifier for the KDL Document Language, a node-based, human-friendly configuration and serialization format.

The parser in this package focuses on parsing documents in a way that allows for format-preserving modifications. This is most useful when working with KDL files maintained by humans.

Try it out

Version 0.3.1 of the @bgotink/kdl package is available on this website via the console of your browser:

kdl exposes the @bgotink/kdl export
jik exposes the @bgotink/kdl/json export

console.log(kdl.parse("node arg").findNodeByName("node").getArgument(0));

Install

yarn add @bgotink/kdl
# or
npm install @bgotink/kdl
# or
pnpm add @bgotink/kdl
# or ...

The examples below assume a Node.js environment. The @bgotink/kdl package works in any javascript runtime, including browsers, as long as it supports relatively modern browser standards. If you're trying to run these samples in a non-Node.js runtime, replace the import for assert with something equivalent in your runtime.

import {parse, format} from "@bgotink/kdl";
import assert from "node:assert/strict";

const doc = parse(String.raw`
	node "value" #"other value"# 2.0 4 #false \
			#null -0 {
		child; "child too"
	}
`);

doc.nodes[0].children.nodes[0].entries.push(
	parse(
		String.raw`/-lorem="ipsum" \
			dolor=#true`,
		{as: "entry"},
	),
);

assert.equal(
	format(doc),
	String.raw`
	node "value" #"other value"# 2.0 4 #false \
			#null -0 {
		child /-lorem="ipsum" \
			dolor=#true; "child too"
	}
`,
);

JSON-in-KDL (JiK)

This package exports function from @bgotink/kdl/json to parse and stringify KDL documents as JSON-in-KDL (JiK) 4.0.0. For information on the format, see the JiK 4.0.0 specification.

import {parse} from "@bgotink/kdl/json";
import assert from "node:assert/strict";

assert.deepEqual(
	parse(
		String.raw`
			- {
				prop #false
				otherProp {
					- 0
					- 2
					- 4
				}
			}
		`,
	),
	{
		prop: false,
		otherProp: [0, 2, 4],
	},
);

There are four functions:

parse and stringify turn text into the encoded JSON value and back. These functions are useful for encoding/decoding entire JiK files.
toJson and fromJson turn KDL nodes into the encoded JSON value and back. These functions give more fine-grained control over the output, and can be used for e.g. encoding/decoding a JiK node embedded in a KDL document.

(De)Serialization

The package contains utilities for (de)serialization in @bgotink/kdl/dessert.

import {parse} from "@bgotink/kdl/dessert";
import assert from "node:assert/strict";

assert.deepEqual(
	parse(
		String.raw`
			name "my-package"
			
			dependency "@bgotink/kdl" range="^0.2.0"
			dependency "prettier" range="^3" dev=#true
		`,
		(ctx) => {
			const dependencies = {};
			const devDependencies = {};

			const name = ctx.child.single.required("name", (ctx) =>
				ctx.argument.required("string"),
			);

			for (const dependency of ctx.children("dependency", (ctx) => ({
				name: ctx.argument.required("string"),
				range: ctx.property("range", "string") ?? "*",
				dev: ctx.property("dev", "boolean") ?? false,
			}))) {
				(dependency.dev ? devDependencies : dependencies)[dependency.name] =
					dependency.range;
			}

			return {name, dependencies, devDependencies};
		},
	),
	{
		name: "my-package",
		dependencies: {
			"@bgotink/kdl": "^0.2.0",
		},
		devDependencies: {
			prettier: "^3",
		},
	},
);

KDL v1

The package exports two functions from @bgotink/kdl/v1-compat to support documents written in KDL v1:

parseWithoutFormatting parses a KDL v1 document without storing any formatting information. If the resulting document is passed to format(), a fresh KDL v2 document will be produced and any comments or formatting added by the original document's author will be lost.
parseAndTransform parses a KDL v1 document and transforms all linked formatting information to turn it into a valid KDL v2 document. If the resulting document is passed to format(), the resulting string will be the same document but in KDL v2 syntax. It will include all comments and formatting applied by the original document's author.

These functions embed the 0.1.x version of this package as parser for KDL v1. This old parser is many times larger and a lot slower than the parser written for KDL v2. As such, it is advised to lazy-load the KDL v1 support only if needed. For example, if you want to read configuration that can be KDL v2 or v1:

import {readFile} from "node:fs/promises";
import {parse} from "@bgotink/kdl";

export async function loadConfiguration(path) {
	const content = await readFile(path);

	try {
		return parse(content);
	} catch (e) {
		const {parseWithoutFormatting} = await import("@bgotink/kdl/v1-compat");
		try {
			return parseWithoutFormatting(content);
		} catch (e2) {
			throw new AggregateError(
				[e, e2],
				`Failed to parse configuration at ${path}`,
			);
		}
	}
}

Quirks

This package turns KDL documents into JavaScript objects and vice versa. It is therefore limited by the JavaScript language.

Properties

Multiple properties with the same name are allowed. All duplicated will be preserved, meaning those documents will correctly round-trip. When using node.getProperty()/node.getProperties()/node.getPropertyEntry(), the last property with that name's value will be returned, effectively shadowing any earlier duplicates. Using node.getPropertyEntries()/node.entries does expose the shadowed duplicates, leaving it up to the caller to handle these. Passing the node through clearFormat() removes these shadowed duplicates.

Numbers

JavaScript stores all numbers as 64-bit IEEE 754 floating point numbers. This limits what integer values can be used safely. These limits are lower than you might expect if you're used to working in environments that have a separate 64-bit integer data type.

The original representation of parsed numbers is retained, unless clearFormat is called on the value or any entry/node/document containing the value.

License

This package is licensed under the MIT license, which can be found in LICENSE.md.

The test suite at test/upstream is part of the KDL specification and is available under the Creative Commons Attribution-ShareAlike 4.0 International License.

Parsing KDL

The parse(text[, options]) function parses a KDL document. The text can be passed in as string, Node.js buffer, TypedArray, ArrayBuffer, or DataView.

You can pass the as option to make the function parse something different from a KDL document:

import {parse, Document, Node} from "@bgotink/kdl";

assert(
	parse(
		String.raw`
			node Lorem Ipsum
		`,
	) instanceof Document,
);

assert(
	parse(
		String.raw`
			node Lorem Ipsum
		`,
		{as: "node"},
	) instanceof Node,
);

Locations

Setting the storeLocations option to true makes location information available in the getLocation function.

Two ways of tracking columns

By default the columns in error messages and in getLocation results are tracked by code point. This means that characters that span multiple code points will move the column forward quite a bit. For example: 😅 is a single code point but 🏳️‍🌈 consists of four code points.

Setting the graphemeLocations option to true instead track columns by grapheme. A grapheme is what we humans perceive as a single character. The pride flag that consists of four code points is a single grapheme.

Tracking by code points is the default for the simple reason that it seems to match how columns are tracked in editors like VS Code or Zed. There's also a 6.5x speed difference between the two methods, but even with graphemeLocations enabled the parser succeeds in parsing thousands of documents per second.

Quirks

This package turns KDL documents into JavaScript objects and vice versa. It is therefore limited by the JavaScript language.

Properties

Numbers

The original representation of parsed numbers is retained, unless clearFormat is called on the value or any entry/node/document containing the value.

Formatting KDL

The format(value) function turns a document, node, entry, identifier, value, or tag into a KDL string representing that value.

The KDL DOM classes contain not just their values, but also any information on whitespace and comments required to format the element. This makes this package good at manipulating KDL files maintained by humans, as it supports making modifications with as few changes to the file as possible.

import {parse, format} from "@bgotink/kdl";

const doc = parse(
	String.raw`
		node "value" #"other value"# 2.0 4 #false \
				#null -0 {
			child; "child too"
		}
	`,
);

doc.nodes[0].children.nodes[0].entries.push(
	parse(
		String.raw`/-lorem="ipsum" \
				dolor=#true`,
		{as: "entry"},
	),
);

assert.equal(
	format(doc),
	String.raw`
		node "value" #"other value"# 2.0 4 #false \
				#null -0 {
			child /-lorem="ipsum" \
				dolor=#true; "child too"
		}
	`,
);

All KDL DOM elements that wrap simple values—i.e. Value, Identifier, and Tag—have an optional representation property that declares how its value is to be formatted. This property is set for all parsed elements and ensures that formatting the element results in as few changes as possible. Take care when changing these values, as the representation is not validated when formatting the element.

// Do not do this!
import {Entry, Identifier, Node, Value, format} from "@bgotink/kdl";

const node = new Node(new Identifier("real_name"));
node.name.representation = "fake_name";

const entry = new Entry(new Value(42), new Identifier("property"));
entry.name.representation = "something_else";
entry.value.representation = "false";

assert.equal(format(node), "fake_name something_else=false");

Instances of Document, Node, Entry, and Tag, store information about whitespace. Just like the representation, these fields are not validated when formatting the DOM elements.

Reset

The clearFormat(value) function removes any and all formatting from a document, node, entry, value, identifier, or tag.

JSON-in-KDL

The @bgotink/kdl/json package export exposes functions to handle JSON-in-KDL, aka JiK. There are two families of functions:

The parse and stringify functions are built to be a drop-in replacement for the JSON.parse and JSON.stringify functions. These functions work well for managing entire JiK files.

The toJson and fromJson functions allow for more fine-grained control. There are extra options that support some none-standard JiK behaviour. Most importantly they work with a KDL Node, which makes these functions support JiK nodes embedded into regular a KDL document.

All of these functions throw an InvalidJsonInKdlError if they encounter invalid JiK content.

Compabitility with KDL v1

The parse function in this package only supports KDL v2 documents. If your program wants to be compatible with KDL v1 as well, this package exposes an extra entry point: @bgotink/kdl/v1-compat.

Loading compatibility code

The compatibility endpoint is a lot larger and slower than the regular @bgotink/kdl endpoint. Programs that want to provide compatiblity are therefore encouraged to lazy-load the compatibility code, for example:

import {readFile} from "node:fs/promises";
import {parse} from "@bgotink/kdl";

export async function loadConfiguration(path) {
	const content = await readFile(path);

	try {
		return parse(content);
	} catch (e) {
		const {parseWithoutFormatting} = await import("@bgotink/kdl/v1-compat");
		try {
			return parseWithoutFormatting(content);
		} catch (e2) {
			throw new AggregateError(
				[e, e2],
				`Failed to parse configuration at ${path}`,
			);
		}
	}
}

Numbers

Here's a comparison of @bgotink/kdl and @bgotink/kdl/v1-compat in numbers:

what	`@bgotink/kdl`	`@bgotink/kdl/v1-compat`
size	81,920 bytes	552,783 bytes
size (bundled + minified)	24,593 bytes	261,295 bytes
size (bundled + minified + zipped)	7,489 bytes	63,315 bytes
benchmark	27k docs/sec	17k docs/sec

Two compatibility functions

The compatibility endpoint exposes two functions that parse a KDL v1 text into a Document. Even though both functions yield documents that are functionally equivalent, they serve very different purposes:

parseWithoutFormatting reads the KDL v1 text without storing any formatting, whitespace, comments, etc. The resulting document can be used everywhere a regular KDL v2 document can be used, but stringifying the document via format() will throw out any comments and formatting its author added.
parseAndTransform reads the KDL v1 text and transforms any formatting, whitespace, comments, etc. into their KDL v2 equivalent. The resulting document can safely be stringified via format() to overwrite the original KDL v1 text to automate the migration from KDL v1 to KDL v2.

(De)Serialization Tools

The @bgotink/kdl/dessert package export exposes tools for helping with (de)serializing KDL nodes/documents into a more useful JavaScript structure. The basic tools don't enforce any specific programming paradigm, e.g. enforcing the use of class, but some of the higher level tools exposed do take a more opinionated approach.

The name "dessert" comes from DESerialization & SERialization Tools. Also, who doesn't like dessert?

Deserialization

There is one main deserialization function: deserialize takes a KDL node or document and a deserializer and runs the given node or document through the deserializer. A second function parse takes a KDL document text, parses it into a KDL document and then runs it through deserialize.

There are three types of deserializer:

A function that takes a deserialization context and returns the deserialized value,
An object with a deserialize function on it that takes a deserialization context and returns the deserialized value, or
An object with a deserializeFromNode function on it that takes a KDL node and returns the deserialized value.

A deserialization context is an object with a bunch of functions on it to help extract arguments, properties, and children from the node.

Here's an example of a deserializer using functions:

import type {DeserializationContext} from "@bgotink/kdl/dessert";

type Tree = {value: number; left?: Tree; right?: Tree};

function treeDeserializer(ctx: DeserializationContext): Tree {
	return {
		value: ctx.argument.required("number"),
		left: ctx.child.single("left", treeDeserializer),
		right: ctx.child.single("right", treeDeserializer),
	};
}

export function readTree(node: Node): Tree {
	return deserialize(node, treeDeserializer);
}

and here's that same deserializer as a class:

import type {DeserializationContext} from "@bgotink/kdl/dessert";

class Tree {
	static deserialize(ctx: DeserializationContext): Tree {
		return new Tree(
			ctx.argument.required("number"),
			ctx.child.single("left", Tree),
			ctx.child.single("right", Tree),
		);
	}

	constructor(
		readonly value: number,
		readonly left?: Tree,
		readonly right?: Tree,
	) {}
}

export function readTree(node: Node): Tree {
	return deserialize(node, Tree);
}

Both of these deserializers effortlessly turn the following KDL node into a tree structure

root 10 {
	left 5 {
		left 3
		right 2
	}
	right 5 {
		left 1
		right 4 {
			left 1
			right 3 {
				left 2
				right 1
			}
		}
	}
}

Serialization

There is one serialization function: serialize takes a node name and a serializer and creates a node with that name and runs it through the serializer. Serializers can be parameterized, allowing for serializers to be shared across multiple values.

There are three types of serializer:

A function that takes a serialization context,
An object with a serialize function on it that takes a serialization context, or
An object with a serializeToNode function on it that returns a KDL node.

All three of these can be parameterized.

Here's an example of a parameterized serializer for the Tree type from either of the two deserialization examples above.

function treeSerializer(ctx: SerializationContext, tree: Tree) {
	ctx.argument(tree.value);

	if (tree.left) {
		ctx.child("left", treeSerializer, tree.left);
	}
	if (tree.right) {
		ctx.child("right", treeSerializer, tree.right);
	}
}

export function writeTree(tree: Tree): Node {
	return serialize("root", treeDeserializer, tree);
}

or we could extend the Tree class from the class-based sample with a serialize method:

class Tree {
	// ... see the Tree class in the deserialize example

	serialize(ctx: SerializationContext) {
		ctx.argument(this.value);

		if (this.left) {
			ctx.child("left", this.left);
		}
		if (this.right) {
			ctx.child("right", this.right);
		}
	}
}

export function writeTree(tree: Tree): Node {
	return serialize("root", tree);
}

Preserving comments and formatting

Programs that make modifications to human-edited files might want to preserve comments and formatting when making these changes. The @bgotink/kdl/dessert API supports this by allowing programs to link an object being serialized using a serialization context with its deserialization context. This process takes two steps:

Store the DeserializationContext somewhere
Pass call source on the SerializationContext at the beginning of the serialize function and pass it the stored DeserializationContext

It is important this call to source happens before you make any other calls to any function on the SerializationContext. Calling it later in the serializer will cause an error.

Here's an example of what this looks like in the class-based Tree example from above:

class Tree {
	// ... see the Tree class in the deserialize and serialize examples

	// We store the DeserializationContext
	static deserialize(ctx: DeserializationContxt) {
		const tree = new Tree(
			ctx.argument.required("number"),
			ctx.child.single("left", Tree),
			ctx.child.single("right", Tree),
		);
		tree.#deserializationCtx = ctx;
		return tree;
	}

	#deserializationCtx?: DeserializationContxt;

	// ...

	serialize(ctx: SerializationContext) {
		ctx.source(this.#deserializationCtx);

		// keep the rest of the original serialize function here
	}
}

If we now pass in this tree

root 10 {
	left 5 { /- left 0; right 5 }
	right 5 { left 1; right 4 }
}

and run

function modify(node: Node): Node {
	const tree = readTree(node); // see deserialization example for this functino

	tree.right.left.value++;
	tree.right.value++;
	tree.value++;

	return writeTree(tree);
}

the resulting node will be

root 11 {
	left 5 { /- left 0; right 5 }
	right 6 { left 2; right 4 }
}

instead of creating a fresh KDL document with default formatting, which would look more like this:

root 11 {
	left 5 {
		right 5
	}
	right 6 {
		left 2
		right 4
	}
}

Deserializers and serializers that implement the deserializeFromNode and serializeToNode respectively are responsible for copying any comments and formatting from the original node being deserialized to the final serialized node.

index

Index

Classes

Interfaces

BOM

A Byte-Order Mark at the start of a document

Properties

text

text: string

The BOM text, i.e. '\ufeff'.

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"value"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"value"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"identifier"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"identifier"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"entry"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"entry"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"node"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"node"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`