Codegen in Hare v2 November 26, 2022 on Drew DeVault's blog

I spoke about code generation in Hare back in May when I wrote a tool for generating ioctl numbers. I wrote another code generator over the past few weeks, and it seems like a good time to revisit the topic on my blog to showcase another approach, and the improvements we’ve made for this use-case.

In this case, I wanted to generate code to implement IPC (inter-process communication) interfaces for my operating system. I have designed a DSL for describing these interfaces — you can read the grammar here. This calls for a parser, which is another interesting topic for Hare, but I’ll set that aside for now and focus on the code gen. Assume that, given a file like the following, we can parse it and produce an AST:

namespace hello;

interface hello {
	call say_hello() void;
	call add(a: uint, b: uint) uint;
};

The key that makes the code gen approach we’re looking at today is the introduction of strings::template to the Hare standard library. This module is inspired by a similar feature from Python, string.Template. An example of its usage is provided in Hare’s standard library documentation:

const src = "Hello, $user! Your balance is $$$balance.\n";
const template = template::compile(src)!;
defer template::finish(&template);
template::execute(&template, os::stdout,
	("user", "ddevault"),
	("balance", 1000),
)!; // "Hello, ddevault! Your balance is $1000.

Makes sense? Cool. Let’s see how this can be applied to code generation. The interface shown above compiles to the following generated code:

// This file was generated by ipcgen; do not modify by hand
use errors;
use helios;
use rt;

def HELLO_ID: u32 = 0xC01CAAC5;

export type fn_hello_say_hello = fn(object: *hello) void;
export type fn_hello_add = fn(object: *hello, a: uint, b: uint) uint;

export type hello_iface = struct {
	say_hello: *fn_hello_say_hello,
	add: *fn_hello_add,
};

export type hello_label = enum u64 {
	SAY_HELLO = HELLO_ID << 16u64 | 1,
	ADD = HELLO_ID << 16u64 | 2,
};

export type hello = struct {
	_iface: *hello_iface,
	_endpoint: helios::cap,
};

export fn hello_dispatch(
	object: *hello,
) void = {
	const (tag, a1) = helios::recvraw(object._endpoint);
	switch (rt::label(tag): hello_label) {
	case hello_label::SAY_HELLO =>
		object._iface.say_hello(
			object,
		);
		match (helios::reply(0)) {
		case void =>
			yield;
		case errors::invalid_cslot =>
			yield; // callee stored the reply
		case errors::error =>
			abort(); // TODO
		};
	case hello_label::ADD =>
		const rval = object._iface.add(
			object,
			a1: uint,
			rt::ipcbuf.params[1]: uint,
		);
		match (helios::reply(0, rval)) {
		case void =>
			yield;
		case errors::invalid_cslot =>
			yield; // callee stored the reply
		case errors::error =>
			abort(); // TODO
		};
	case =>
		abort(); // TODO
	};
};

Generating this code starts with the following entry-point:

// Generates code for a server to implement the given interface.
export fn server(out: io::handle, doc: *ast::document) (void | io::error) = {
	fmt::fprintln(out, "// This file was generated by ipcgen; do not modify by hand")!;
	fmt::fprintln(out, "use errors;")!;
	fmt::fprintln(out, "use helios;")!;
	fmt::fprintln(out, "use rt;")!;
	fmt::fprintln(out)!;

	for (let i = 0z; i < len(doc.interfaces); i += 1) {
		const iface = &doc.interfaces[i];
		s_iface(out, doc, iface)?;
	};
};

Here we start with some simple use of basic string formatting via fmt::fprintln. We see some of the same approach repeated in the meatier functions like s_iface:

fn s_iface(
	out: io::handle,
	doc: *ast::document,
	iface: *ast::interface,
) (void | io::error) = {
	const id: ast::ident = [iface.name];
	const name = gen_name_upper(&id);
	defer free(name);

	let id: ast::ident = alloc(doc.namespace...);
	append(id, iface.name);
	defer free(id);
	const hash = genhash(&id);

	fmt::fprintfln(out, "def {}_ID: u32 = 0x{:X};\n", name, hash)!;

Our first use of strings::template appears when we want to generate type aliases for interface functions, via s_method_fntype. This is where some of the trade-offs of this approach begin to present themselves.

const s_method_fntype_src: str =
	`export type fn_$iface_$method = fn(object: *$object$params) $result;`;
let st_method_fntype: tmpl::template = [];

@init fn s_method_fntype() void = {
	st_method_fntype= tmpl::compile(s_method_fntype_src)!;
};

fn s_method_fntype(
	out: io::handle,
	iface: *ast::interface,
	meth: *ast::method,
) (void | io::error) = {
	assert(len(meth.caps_in) == 0); // TODO
	assert(len(meth.caps_out) == 0); // TODO

	let params = strio::dynamic();
	defer io::close(&params)!;
	if (len(meth.params) != 0) {
		fmt::fprint(&params, ", ")?;
	};
	for (let i = 0z; i < len(meth.params); i += 1) {
		const param = &meth.params[i];
		fmt::fprintf(&params, "{}: ", param.name)!;
		ipc_type(&params, &param.param_type)!;

		if (i + 1 < len(meth.params)) {
			fmt::fprint(&params, ", ")!;
		};
	};

	let result = strio::dynamic();
	defer io::close(&result)!;
	ipc_type(&result, &meth.result)!;

	tmpl::execute(&st_method_fntype, out,
		("method", meth.name),
		("iface", iface.name),
		("object", iface.name),
		("params", strio::string(&params)),
		("result", strio::string(&result)),
	)?;
	fmt::fprintln(out)?;
};

The simple string substitution approach of strings::template prevents it from being as generally useful as a full-blown templating engine ala jinja2. To work around this, we have to write Hare code which does things like slurping up the method parameters into a strio::dynamic buffer where we might instead reach for something like {% for param in method.params %} in jinja2. Once we have prepared all of our data in a format suitable for a linear string substitution, we can pass it to tmpl::execute. The actual template is stored in a global which is compiled during @init, which runs at program startup. Anything which requires a loop to compile, such as the parameter list, is fetched out of the strio buffer and passed to the template.

We can explore a slightly different approach when we generate this part of the code, back up in the s_iface function:

export type hello_iface = struct {
	say_hello: *fn_hello_say_hello,
	add: *fn_hello_add,
};

To output this code, we render several templates one after another, rather than slurping up the generated code into heap-allocated string buffers to be passed into a single template.

const s_iface_header_src: str =
	`export type $iface_iface = struct {`;
let st_iface_header: tmpl::template = [];

const s_iface_method_src: str =
	`	$method: *fn_$iface_$method,`;
let st_iface_method: tmpl::template = [];

@init fn s_iface() void = {
	st_iface_header = tmpl::compile(s_iface_header_src)!;
	st_iface_method = tmpl::compile(s_iface_method_src)!;
};

// ...

tmpl::execute(&st_iface_header, out,
	("iface", iface.name),
)?;
fmt::fprintln(out)?;

for (let i = 0z; i < len(iface.methods); i += 1) {
	const meth = &iface.methods[i];
	tmpl::execute(&st_iface_method, out,
		("iface", iface.name),
		("method", meth.name),
	)?;
	fmt::fprintln(out)?;
};

fmt::fprintln(out, "};\n")?;

The remainder of the code is fairly similar.

strings::template is less powerful than a more sophisticated templating system might be, such as Golang’s text/template. A more sophisticated templating engine could be implemented for Hare, but it would be more challenging — no reflection or generics in Hare — and would not be a great candidate for the standard library. This approach hits the sweet spot of simplicity and utility that we’re aiming for in the Hare stdlib. strings::template is implemented in a single ~180 line file.

I plan to continue polishing this tool so I can use it to describe interfaces for communications between userspace drivers and other low-level userspace services in my operating system. If you have any questions, feel free to post them on my public inbox, or shoot them over to my new fediverse account. Until next time!

Have a comment on one of my posts? Start a discussion in my public inbox by sending an email to ~sircmpwn/public-inbox@lists.sr.ht [mailing list etiquette]

Articles from blogs I read Generated by openring

Share your feedback about developing with Go

Help shape the future of Go by sharing your thoughts via the Go Developer Survey

via The Go Blog January 18, 2023

Status update, January 2023

Hi all! This month’s status update will be lighter than usual: I’ve been on leave for a while at the end of December. To make up for this, I have some big news: we’ve released Sway 1.8! This brings a whole lot of improvements from wlroots 0.16, as well as som…

via emersion January 16, 2023

SourceHut will (not) blacklist the Go module mirror

Update 2023-01-31: Russ Cox of the Go team reached out to us to address this problem. After some discussion, an acceptable plan was worked out. The Go team is working on deploying an update to the “go” tool to add a -reuse flag, which should substantially re…

via Blogs on Sourcehut January 9, 2023