Adventures in A11y: Trying Out Talon & Cursorless, part 0

“Shay, you mentioned some number of episodes ago that the army took your mouse for a while. What would you do if they took your keyboard, as well?”

This post is starting out by me typing it on my keyboard. However, if things go well, by the end of this post some part should be typed out with my voice.

Every part that is in between lines (like this one) is actually typed out with Cursorless and with my voice, not my keyboard.I’ll try to keep some of the mistakes in which is why these parts might be slightly weirdly formatted or even have some mistakes.

what you’re reading right now has actually been written at the end of the post after I’ve learned everything that is detailed out below.

How did we get here?

On a recent Cup o’ Go episode I interviewed Josh Bleecher Snyder, and he mentioned that he was using Cursorless to edit his code, since he was dealing with some wrist pain. I was intrigued, and decided to give it a try. This post is here to document my experience, share it with Josh and the Cursorless team to help them with feedback, and perhaps to help you get started, as well! Investing time into a11y related projects is an important thing to do, and I’m happy to do take a few hours out of my weekend to help out.

Installation

I’m installing on a Macbook Pro M1. Here’s the documentation link. A few notes from my installation process:

Talon

After installing Talon, you need to open it and do a few things. Basically, you have to go into the getting started guide and follow everything that’s written there, which isn’t mentioned in the Cursorless docs. Here are a few gotchas that I dealt with.

1. Give it permissions for accessibility settings. You’ll get the normal MacOS pop up.
2. You need to open Talon and accept the EULA.

Only then can you move to the next step, since until you do that, you won’t have the ~/.talon/user directory.

You also need to configure a speech engine, see below.

talonhub/community (also known as knausj_talon)

Installing talonhub/community (also known as knausj_talon) is pretty simple; just run

cd ~/.talon/user
git clone https://github.com/talonhub/community community

VSCode + extension

I already have VSCode, and the extensions are easy to install. As usual, we try to do things here with the keyboard only. So to open the extensions section of VSCode, open the command palette (with ⌘ + P) and write:

And then you’re good to go! Just search for the two extensions needed, Talon and Cursorless, and install them.

The documentation then says “restart Talon”. I wish they gave me a command to run, but I guesstimated that right-clicking it in the command bar, quitting, and then running it again would do the trick. And it did.

Let’s try to go Cursorless

I’m still typing with my hands at this point, but my VSCode looks all funny:

I’ve been following this tutorial video:

The tutorial assumes that I have installed everything (which I did), and that I’m also familiar with the “Talon Alphabet”. I am NOT, but the video said:

“With Talon listening, you can just say “help alphabet”.

Really? I’ve decided to record my first time trying this out. Exciting.

AAAAAAnd nothing happens. What gives? I read through Talon’s Getting Started guide and it mentions configuring a speech recognition engine. Looking at my Talon instance, it seems like there’s no speech engine loaded.

After loading one, I pressed record, and this is my real reaction trying out Talon for the first time:

This was… WILD. I didn’t expect it to do so many things: it messed up my OBS, opened GitHub (for some reason), and opened a phone menu. I didn’t learn about the all-important “Talon Sleep” and “Talon Wake” commands yet, but I found them in the Getting Started guides and used them to stop the madness.

Now that I’ve learned my lesson, I’ll watch the tutorial video all the way through before continuing.

I’ve watched the tutorial, so now, finally, time to try out Cursorless itself for writing this blog!

so I’m just gonna start talking and see if this thing is typing.l(l)

OK, so that went… Poorly. 😅 Check this out:

Here’s what happened:

1. I remembered talon wake.
2. I remembered take, but I wasn’t sure how to select capital letters.
3. I failed opening the Cursorless cheat sheet, instead typing chichie.
4. Now that I (mistakenly) had some lowercase letters, I could try out chuck.
5. The I started talking, and Cursorless actually typed what I said!
6. But then it went off the rails for some reason and selected random sections, at which point I called it and just said talon sleep.

Also, there was another thing bothering me which you can see in the video. I have two monitors at home: the laptop and a big screen. The issue was that I’m currently using laptop monitor, but Talon opened help and the printed the subtitles into the big monitor.

I got into the Slack for some help and immediately someone answered! Thanks, Timotimo:

Installed the Talon HUD

Following Timotimo’s advice, I decided to install the Talon HUD to help with my learning and understanding.

Unlike what the GitHub README suggests (unzip), I decided to install using git clone into the ~/.talon/user directory. So just run the following commands:

cd ~/.talon/user
git clone git@github.com:chaosparrot/talon_hud.git

Immediately after that, I got this on my screen:

And trying understand how to work with it again:

3

!*.

And… it didn’t go very well 😅 Check out the videos here:

The Talon HUD walkthrough

I had an issue with Talon not starting up, so I restarted it, and got the Talon HUD walkthrough:

Following the interactive tutorial was GREAT, and it was a much better way of learning voice commands than trying to brute-force it the way I did. I did encounter some bugs which doing the walkthrough:

So I reported them, like the good OSS citizen that I am! Bug #1 and bug #2.

Finally, I finished the initial walkthrough! It felt really good. I’ll definitely try the rest of the walkthroughs later!

Contributing the the Talon Speech Dataset

From the Slack community, I’ve found this message:

Reminder: Remember to record: https://speech.talonvoice.com/

Curiosity got the better of me, and I clicked the link. And it’s another way to contribute to the project.

My unique identifier is adf35d69-1a27-41af-9d20-2427b00767cc if you want to look for my contributions in the public dataset: I’ve contributed four recordings.

Final attempt and summary

So, I really wanted to type this part with Talon. But I’m not there yet. As a compromise, I’m marking this post as a part 0, hoping to come up with part 1 soon!

I think that investing my time and effort into Talon + Cursorless would give me great insight into a11y in general and is