`@bgotink/kdl`

This package contains a parser and stringifier for the KDL Document Language, a node-based, human-friendly configuration and serialization format.

The parser in this package focuses on parsing documents in a way that allows for format-preserving modifications. This is most useful when working with KDL files maintained by humans.

Try it out

Version 0.4.0-next.0 of the @bgotink/kdl package is available on this website via the console of your browser:

kdl exposes the @bgotink/kdl export
jik exposes the @bgotink/kdl/json export

console.log(kdl.parse("node arg").findNodeByName("node").getArgument(0));

Install

yarn add @bgotink/kdl
# or
npm install @bgotink/kdl
# or
pnpm add @bgotink/kdl
# or ...

The examples below assume a Node.js environment. The @bgotink/kdl package works in any javascript runtime, including browsers, as long as it supports relatively modern browser standards. If you're trying to run these samples in a non-Node.js runtime, replace the import for assert with something equivalent in your runtime.

import {parse, format} from "@bgotink/kdl";
import assert from "node:assert/strict";

const doc = parse(String.raw`
	node "value" #"other value"# 2.0 4 #false \
			#null -0 {
		child; "child too"
	}
`);

doc.nodes[0].children.nodes[0].entries.push(
	parse(
		String.raw`/-lorem="ipsum" \
			dolor=#true`,
		{as: "entry"},
	),
);

assert.equal(
	format(doc),
	String.raw`
	node "value" #"other value"# 2.0 4 #false \
			#null -0 {
		child /-lorem="ipsum" \
			dolor=#true; "child too"
	}
`,
);

JSON-in-KDL (JiK)

This package exports function from @bgotink/kdl/json to parse and stringify KDL documents as JSON-in-KDL (JiK) 4.0.0. For information on the format, see the JiK 4.0.0 specification.

import {parse} from "@bgotink/kdl/json";
import assert from "node:assert/strict";

assert.deepEqual(
	parse(
		String.raw`
			- {
				prop #false
				otherProp {
					- 0
					- 2
					- 4
				}
			}
		`,
	),
	{
		prop: false,
		otherProp: [0, 2, 4],
	},
);

There are four functions:

parse and stringify turn text into the encoded JSON value and back. These functions are useful for encoding/decoding entire JiK files.
toJson and fromJson turn KDL nodes into the encoded JSON value and back. These functions give more fine-grained control over the output, and can be used for e.g. encoding/decoding a JiK node embedded in a KDL document.

(De)Serialization

The package contains utilities for (de)serialization in @bgotink/kdl/dessert.

import {parse} from "@bgotink/kdl/dessert";
import assert from "node:assert/strict";

assert.deepEqual(
	parse(
		String.raw`
			name "my-package"
			
			dependency "@bgotink/kdl" range="^0.2.0"
			dependency "prettier" range="^3" dev=#true
		`,
		(ctx) => {
			const dependencies = {};
			const devDependencies = {};

			const name = ctx.child.single.required("name", (ctx) =>
				ctx.argument.required("string"),
			);

			for (const dependency of ctx.children("dependency", (ctx) => ({
				name: ctx.argument.required("string"),
				range: ctx.property("range", "string") ?? "*",
				dev: ctx.property("dev", "boolean") ?? false,
			}))) {
				(dependency.dev ? devDependencies : dependencies)[dependency.name] =
					dependency.range;
			}

			return {name, dependencies, devDependencies};
		},
	),
	{
		name: "my-package",
		dependencies: {
			"@bgotink/kdl": "^0.2.0",
		},
		devDependencies: {
			prettier: "^3",
		},
	},
);

KDL v1

The package exports two functions from @bgotink/kdl/v1-compat to support documents written in KDL v1:

parse parses a KDL v1 document and transforms all linked formatting information to turn it into a valid KDL v2 document. If the resulting document is passed to format(), the resulting string will be the same document but in KDL v2 syntax. It will include all comments and formatting applied by the original document's author.
parseCompat parses a document that's either KDL v2 or KDL v1 and returns a valid KDL v2 document. This is a helper function that combines the regular parse function with the v1-compat parse function into a single function that supports both formats.

Quirks

This package turns KDL documents into JavaScript objects and vice versa. It is therefore limited by the JavaScript language.

Properties

Multiple properties with the same name are allowed. All duplicated will be preserved, meaning those documents will correctly round-trip. When using node.getProperty()/node.getProperties()/node.getPropertyEntry(), the last property with that name's value will be returned, effectively shadowing any earlier duplicates. Using node.getPropertyEntries()/node.entries does expose the shadowed duplicates, leaving it up to the caller to handle these. Passing the node through clearFormat() removes these shadowed duplicates.

Numbers

JavaScript stores all numbers as 64-bit IEEE 754 floating point numbers. This limits what integer values can be used safely. These limits are lower than you might expect if you're used to working in environments that have a separate 64-bit integer data type.

The original representation of parsed numbers is retained, unless clearFormat is called on the value or any entry/node/document containing the value.

License

This package is licensed under the MIT license, which can be found in LICENSE.md.

The test suite at test/upstream is part of the KDL specification and is available under the Creative Commons Attribution-ShareAlike 4.0 International License.

Parsing KDL

The parse(text[, options]) function parses a KDL document. The text can be passed in as string, Node.js buffer, TypedArray, ArrayBuffer, or DataView.

You can pass the as option to make the function parse something different from a KDL document:

import {parse, Document, Node} from "@bgotink/kdl";

assert(
	parse(
		String.raw`
			node Lorem Ipsum
		`,
	) instanceof Document,
);

assert(
	parse(
		String.raw`
			node Lorem Ipsum
		`,
		{as: "node"},
	) instanceof Node,
);

Errors

The parse function throws InvalidKdlErrors when an error occurs. The parser tries to recover from many errors so it can provide as much feedback as possible in a single run. This greatly helps when using the parser to provide feedback to the human author of a KDL document, as you can provide them with multiple mistakes in a single go instaed of showing only the first error every single time.

A single InvalidKdlError object might describe multiple actual errors in the document. The getDetails() method returns an iterator that yields every KDL error contained within the error. If the error doesn't have any more details, the getDetails() iterator only yields the error itself..

Locations

Setting the storeLocations option to true makes location information available in the getLocation function.

Two ways of tracking columns

By default the columns in error messages and in getLocation results are tracked by code point. This means that characters that span multiple code points will move the column forward quite a bit. For example: 😅 is a single code point but 🏳️‍🌈 consists of four code points.

Setting the graphemeLocations option to true instead track columns by grapheme. A grapheme is what we humans perceive as a single character. The pride flag that consists of four code points is a single grapheme.

Tracking by code points is the default for the simple reason that it seems to match how columns are tracked in editors like VS Code or Zed. There's also a 6.5x speed difference between the two methods, but even with graphemeLocations enabled the parser succeeds in parsing thousands of documents per second.

Quirks

This package turns KDL documents into JavaScript objects and vice versa. It is therefore limited by the JavaScript language.

Properties

Numbers

The original representation of parsed numbers is retained, unless clearFormat is called on the value or any entry/node/document containing the value.

Formatting KDL

The format(value) function turns a document, node, entry, identifier, value, or tag into a KDL string representing that value.

The KDL DOM classes contain not just their values, but also any information on whitespace and comments required to format the element. This makes this package good at manipulating KDL files maintained by humans, as it supports making modifications with as few changes to the file as possible.

import {parse, format} from "@bgotink/kdl";

const doc = parse(
	String.raw`
		node "value" #"other value"# 2.0 4 #false \
				#null -0 {
			child; "child too"
		}
	`,
);

doc.nodes[0].children.nodes[0].entries.push(
	parse(
		String.raw`/-lorem="ipsum" \
				dolor=#true`,
		{as: "entry"},
	),
);

assert.equal(
	format(doc),
	String.raw`
		node "value" #"other value"# 2.0 4 #false \
				#null -0 {
			child /-lorem="ipsum" \
				dolor=#true; "child too"
		}
	`,
);

All KDL DOM elements that wrap simple values—i.e. Value, Identifier, and Tag—have an optional representation property that declares how its value is to be formatted. This property is set for all parsed elements and ensures that formatting the element results in as few changes as possible. Take care when changing these values, as the representation is not validated when formatting the element.

// Do not do this!
import {Entry, Identifier, Node, Value, format} from "@bgotink/kdl";

const node = new Node(new Identifier("real_name"));
node.name.representation = "fake_name";

const entry = new Entry(new Value(42), new Identifier("property"));
entry.name.representation = "something_else";
entry.value.representation = "false";

assert.equal(format(node), "fake_name something_else=false");

Instances of Document, Node, Entry, and Tag, store information about whitespace. Just like the representation, these fields are not validated when formatting the DOM elements.

Reset

The clearFormat(value) function removes any and all formatting from a document, node, entry, value, identifier, or tag.

JSON-in-KDL

The @bgotink/kdl/json package export exposes functions to handle JSON-in-KDL, aka JiK. There are two families of functions:

The parse and stringify functions are built to be a drop-in replacement for the JSON.parse and JSON.stringify functions. These functions work well for managing entire JiK files.

The toJson and fromJson functions allow for more fine-grained control. There are extra options that support some none-standard JiK behaviour. Most importantly they work with a KDL Node, which makes these functions support JiK nodes embedded into regular a KDL document.

All of these functions throw an InvalidJsonInKdlError if they encounter invalid JiK content.

Compabitility with KDL v1

The parse function in this package only supports KDL v2 documents. If your program wants to be compatible with KDL v1 as well, this package exposes an extra entry point @bgotink/kdl/v1-compat.

This entry point exposes two functions: parse and parseCompat. Both functions return a Document that produces valid KDL v2 text, but they differ in what they accept.

parse parses a KDL v1 text
parseCocmpat parses KDL text that's either valid KDL v2 or KDL v1

The parseCompat function first tries to parse the text as a KDL v2 document. If that fails, it tries to parse the text as a KDL v1 document. If both parsers fail, the function throws an AggregateError that contains all errors. The function does not look for a /- kdl-version <version> hint in the text.

(De)Serialization Tools

The @bgotink/kdl/dessert package export exposes tools for helping with (de)serializing KDL nodes/documents into a more useful JavaScript structure. The basic tools don't enforce any specific programming paradigm, e.g. enforcing the use of class, but some of the higher level tools exposed do take a more opinionated approach.

The name "dessert" comes from DESerialization & SERialization Tools. Also, who doesn't like dessert?

Deserialization

There is one main deserialization function: deserialize takes a KDL node or document and a deserializer and runs the given node or document through the deserializer. A second function parse takes a KDL document text, parses it into a KDL document and then runs it through deserialize.

There are three types of deserializer:

A function that takes a deserialization context and returns the deserialized value,
An object with a deserialize function on it that takes a deserialization context and returns the deserialized value, or
An object with a deserializeFromNode function on it that takes a KDL node and returns the deserialized value.

A deserialization context is an object with a bunch of functions on it to help extract arguments, properties, and children from the node.

Here's an example of a deserializer using functions:

import type {DeserializationContext} from "@bgotink/kdl/dessert";

type Tree = {value: number; left?: Tree; right?: Tree};

function treeDeserializer(ctx: DeserializationContext): Tree {
	return {
		value: ctx.argument.required("number"),
		left: ctx.child.single("left", treeDeserializer),
		right: ctx.child.single("right", treeDeserializer),
	};
}

export function readTree(node: Node): Tree {
	return deserialize(node, treeDeserializer);
}

and here's that same deserializer as a class:

import type {DeserializationContext} from "@bgotink/kdl/dessert";

class Tree {
	static deserialize(ctx: DeserializationContext): Tree {
		return new Tree(
			ctx.argument.required("number"),
			ctx.child.single("left", Tree),
			ctx.child.single("right", Tree),
		);
	}

	constructor(
		readonly value: number,
		readonly left?: Tree,
		readonly right?: Tree,
	) {}
}

export function readTree(node: Node): Tree {
	return deserialize(node, Tree);
}

Both of these deserializers effortlessly turn the following KDL node into a tree structure

root 10 {
	left 5 {
		left 3
		right 2
	}
	right 5 {
		left 1
		right 4 {
			left 1
			right 3 {
				left 2
				right 1
			}
		}
	}
}

Serialization

There is one serialization function: serialize takes a node name and a serializer and creates a node with that name and runs it through the serializer. Serializers can be parameterized, allowing for serializers to be shared across multiple values.

There are three types of serializer:

A function that takes a serialization context,
An object with a serialize function on it that takes a serialization context, or
An object with a serializeToNode function on it that returns a KDL node.

All three of these can be parameterized.

Here's an example of a parameterized serializer for the Tree type from either of the two deserialization examples above.

function treeSerializer(ctx: SerializationContext, tree: Tree) {
	ctx.argument(tree.value);

	if (tree.left) {
		ctx.child("left", treeSerializer, tree.left);
	}
	if (tree.right) {
		ctx.child("right", treeSerializer, tree.right);
	}
}

export function writeTree(tree: Tree): Node {
	return serialize("root", treeDeserializer, tree);
}

or we could extend the Tree class from the class-based sample with a serialize method:

class Tree {
	// ... see the Tree class in the deserialize example

	serialize(ctx: SerializationContext) {
		ctx.argument(this.value);

		if (this.left) {
			ctx.child("left", this.left);
		}
		if (this.right) {
			ctx.child("right", this.right);
		}
	}
}

export function writeTree(tree: Tree): Node {
	return serialize("root", tree);
}

Preserving comments and formatting

Programs that make modifications to human-edited files might want to preserve comments and formatting when making these changes. The @bgotink/kdl/dessert API supports this by allowing programs to link an object being serialized using a serialization context with its deserialization context. This process takes two steps:

Store the DeserializationContext somewhere
Pass call source on the SerializationContext at the beginning of the serialize function and pass it the stored DeserializationContext

It is important this call to source happens before you make any other calls to any function on the SerializationContext. Calling it later in the serializer will cause an error.

Here's an example of what this looks like in the class-based Tree example from above:

class Tree {
	// ... see the Tree class in the deserialize and serialize examples

	// We store the DeserializationContext
	static deserialize(ctx: DeserializationContxt) {
		const tree = new Tree(
			ctx.argument.required("number"),
			ctx.child.single("left", Tree),
			ctx.child.single("right", Tree),
		);
		tree.#deserializationCtx = ctx;
		return tree;
	}

	#deserializationCtx?: DeserializationContxt;

	// ...

	serialize(ctx: SerializationContext) {
		ctx.source(this.#deserializationCtx);

		// keep the rest of the original serialize function here
	}
}

If we now pass in this tree

root 10 {
	left 5 { /- left 0; right 5 }
	right 5 { left 1; right 4 }
}

and run

function modify(node: Node): Node {
	const tree = readTree(node); // see deserialization example for this functino

	tree.right.left.value++;
	tree.right.value++;
	tree.value++;

	return writeTree(tree);
}

the resulting node will be

root 11 {
	left 5 { /- left 0; right 5 }
	right 6 { left 2; right 4 }
}

instead of creating a fresh KDL document with default formatting, which would look more like this:

root 11 {
	left 5 {
		right 5
	}
	right 6 {
		left 2
		right 4
	}
}

Deserializers and serializers that implement the deserializeFromNode and serializeToNode respectively are responsible for copying any comments and formatting from the original node being deserialized to the final serialized node.

index

Classes

Interfaces

BOM

Defined in: src/model/whitespace.d.ts:8

A Byte-Order Mark at the start of a document

Properties

text

text: string

Defined in: src/model/whitespace.d.ts:17

The BOM text, i.e. '\ufeff'.

type

type: "bom"

Defined in: src/model/whitespace.d.ts:12

A property to differentiate the different types of whitespace

EscLine

Defined in: src/model/whitespace.d.ts:55

An escaped newline

Properties

text

text: string

Defined in: src/model/whitespace.d.ts:64

The escaped newline

type

type: "line-escape"

Defined in: src/model/whitespace.d.ts:59

A property to differentiate the different types of whitespace

InlineWhitespace

Defined in: src/model/whitespace.d.ts:23

Regular plain old whitespace characters

Properties

text

text: string

Defined in: src/model/whitespace.d.ts:32

The whitespace's text

type

type: "space"

Defined in: src/model/whitespace.d.ts:27

A property to differentiate the different types of whitespace

Location

Defined in: src/parser/token.d.ts:6

A location in the source text

Properties

column

column: number

Defined in: src/parser/token.d.ts:24

Column in the line

The first character of the line is column number 1.

line

line: number

Defined in: src/parser/token.d.ts:18

Line number in the source text

The first line has number 1.

offset

offset: number

Defined in: src/parser/token.d.ts:12

Index of this location in the source text

The first character in the source text has offset 0.

MultilineComment

Defined in: src/model/whitespace.d.ts:70

A multiline comment

Properties

text

text: string

Defined in: src/model/whitespace.d.ts:79

The comment text, including the comment tokens themselves

type

type: "multiline"

Defined in: src/model/whitespace.d.ts:74

A property to differentiate the different types of whitespace

Newline

Defined in: src/model/whitespace.d.ts:40

A single newline

Note a newline can consist of multiple characters: \r\n is a single newline.

Properties

text

text: string

Defined in: src/model/whitespace.d.ts:49

The newline

type

type: "newline"

Defined in: src/model/whitespace.d.ts:44

A property to differentiate the different types of whitespace

SingleLineComment

Defined in: src/model/whitespace.d.ts:85

A single-line comment

Properties

text

text: string

Defined in: src/model/whitespace.d.ts:94

The comment's text, starting at the // and ending with a newline unless the comment ended at the end of the file

type

type: "singleline"

Defined in: src/model/whitespace.d.ts:89

A property to differentiate the different types of whitespace

SlashDashInDocument

Defined in: src/model/whitespace.d.ts:137

A slashdash comment in a document, i.e. a slashdash commented node

Properties

preface

preface: NodeSpace[]

Defined in: src/model/whitespace.d.ts:146

Any whitespace between the slashdash token and the value

type

type: "slashdash"

Defined in: src/model/whitespace.d.ts:141

A property to differentiate the different types of whitespace

value

value: Node

Defined in: src/model/whitespace.d.ts:151

The escaped value

SlashDashInNode

Defined in: src/model/whitespace.d.ts:112

A slashdash comment inside a node, i.e. a slashdash commented argument, property, or child block

Properties

preface

preface: NodeSpace[]

Defined in: src/model/whitespace.d.ts:121

Any whitespace between the slashdash token and the value

type

type: "slashdash"

Defined in: src/model/whitespace.d.ts:116

A property to differentiate the different types of whitespace

value

value: Entry | Document

Defined in: src/model/whitespace.d.ts:126

The escaped value

StoredLocation

Defined in: src/locations.js:5

Stored location of a Document, Entry, Identifier, Node, Tag, or Value.

Properties

end

end: Location

Defined in: src/locations.js:9

The location after the last character

start

start: Location

Defined in: src/locations.js:8

The location of the first character

Token

Defined in: src/parser/token.d.ts:30

A single token in the KDL text

Properties

end

end: Location

Defined in: src/parser/token.d.ts:60

The location after the last character of this token

start

start: Location

Defined in: src/parser/token.d.ts:55

The location of the first character of this token

text

text: string

Defined in: src/parser/token.d.ts:50

The text of this token

This could be computed if you have access to the source text using

sourceText.slice(token.start.offset, token.end.offset)

Type Aliases

LineSpace

LineSpace: BOM | Newline | WS | SingleLineComment

Defined in: src/model/whitespace.d.ts:107

A single plain whitespace item in a document, i.e. before/after/between nodes

NodeSpace

NodeSpace: EscLine | WS

Defined in: src/model/whitespace.d.ts:102

A single plain whitespace item inside of a node, e.g. between two arguments in a node.

Primitive

Primitive<>: string | number | boolean | null

Defined in: src/model/value.js:4

A primitive is any type that can be represented as an argument or property

Type Parameters

Type Parameter

WhitespaceInDocument

WhitespaceInDocument: (LineSpace | SlashDashInDocument)[]

Defined in: src/model/whitespace.d.ts:157

Whitespace in a document, i.e. before/after/between nodes

WhitespaceInNode

WhitespaceInNode: (NodeSpace | SlashDashInNode)[]

Defined in: src/model/whitespace.d.ts:132

Whitespace inside of a node, e.g. between two arguments in a node.

Functions

clearFormat()

clearFormat<T>(v): T

Defined in: src/clear-format.js:123

Type Parameters

Type Parameter	Description
`T` extends `Identifier` \| `Tag` \| `Value` \| `Entry` \| `Node` \| `Document`

Parameters

Parameter	Type	Description
`v`	`T`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"value"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"value"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"identifier"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"identifier"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"entry"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"entry"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`

Parameter	Type
`text`	`string` \| `ArrayBuffer` \| `DataView` \| `Int8Array` \| `Uint8Array` \| `Int16Array` \| `Uint16Array` \| `Int32Array` \| `Uint32Array`
`options`	{ `as`: `"node"`; `graphemeLocations`: `boolean`; `storeLocations`: `boolean`; }
`options.as`	`"node"`
`options.graphemeLocations`?	`boolean`
`options.storeLocations`?	`boolean`