The MouthPad is a hands-free computer input device. For nearly two years, the MouthPad has been available in early access in which we have worked with approximately 150 people with a range of uses for the MouthPad to better understand where it can be impactful. Here we summarize some of those learnings.

The MouthPad enables hands-free computer control by using tongue gestures and, if the user is willing/able, head movements generate mouse HID inputs (we’ll talk more about keyboard inputs shortly). Internally, we refer to these as “tongue control” and “head control”, even though both control modes make use of tongue gestures. Three types of tongue gestures are available as core gestures: (1) touching the trackpad, which reports tongue touch location, (2) tongue press, or pressing the tongue into the roof of the mouth to apply positive pressure and (3) sip, or applying negative pressure the way you would sip a beverage. In both modes, press generates a left click and sip generates a right click. In tongue control mode, the tongue touch location controls a virtual joystick, moving the cursor in a particular direction and speed depending on the touch location. In head control mode, the user’s head movements move the cursor and the tongue touch location controls virtual buttons to enable scrolling vertically and horizontally.

For simplicity in this first discussion, we will begin by reporting results for head control users. Generally our users report that head control is faster than tongue control, in much the same way that moving a mouse is faster than pointing your cursor with a joystick [34]. We estimate that about a quarter of MouthPad users are primarily tongue control users. In some cases, head control mode is not physically possible for a user. Or, if a user wants to use the device while lying down on their back, head control works but may be uncomfortable as it requires moving your head against a surface like a pillow or a bed. Or, if a user is in a bumpy vehicle head control is noisy due to vibrations of the road.

Point-and-click performance

Point and left click is the cornerstone of being able to do anything on your computer. As part of onboarding to the MouthPad, users were suggested to perform a 1-minute grid click selection task to assess their MouthPad settings, and to record their scores on our game leaderboard if they wanted. This simple game is nominally the same as Neuralink’s version for comparison, and versions of the same game are commonly used in assessments of brain-computer interfaces [20].

58 people with hand impairments completed the 1-minute grid selection game and recorded scores. Hand impairment was due to a variety of conditions summarized in Table 1. “Other” represents a combination users who either declined to state their precise hand impairment as well as users whose impairment is known but unique in the cohort.

Table 1. Conditions leading to hand impairment in 58 MouthPad users who completed the grid assessment task.

In addition, 23 nominally healthy individuals also participated in MouthPad^ tasks. This included a mix of Augmental staff, external assistive technology professionals (ATPs) who work in the clinical setting to prescribe assistive devices to people with hand impairments, and HCI researchers developing hands-free computer interfaces. These users are highly experienced with alternative technologies for computer interaction, possibly leading them to perform better than a typical computer user.

Point and click performance in this task is measurable in bits per second (bits/sec) - each click metaphorically “communicates” $\log_2(\text{number of targets})$ bits. So in our 30x30 grid, each click communicates 9.81 raw bits. Errors like misclicks communicate the wrong raw bits. Like in any communication channel, the effective bitrate requires accounting for these errors. One way to do so is:

$$ \mathrm{NTPM} = \#\text{ correct clicks} - \#\text{ misclicks} $$

$$ \text{Selection rate} = \frac{\log_2(\#\text{ of targets}) \cdot \mathrm{NTPM}}{\text{time}} \; \mathrm{bits/sec} $$

The concept of “net trials per minute” (NTPM) is on the conservative side, as no “partial credit” is given for misclicks adjacent to the target vs further away [29]. While this quantifies performance, it is difficult to translate the number to an intuitive functional capability. Just like how the door-to-door internet service provider salesperson needs to translate the internet bitrate into the number of simultaenous streaming videocalls it supports, a video is worth a thousand words to understand what a 4 bits/sec point-and-click can poentially unlokc. Below is the onboarding of one MouthPad user, who achieves 4.25 bits/sec in his first attempt on the grid selection task in his first hour of using the MouthPad

https://www.linkedin.com/posts/augmental-tech_mouthpad-onboarding-the-first-hour-activity-7274831271933939712-N6Og?utm_source=share&utm_medium=member_desktop&rcm=ACoAAATDqiQBFZ49UaaDU4qmJkNYXlD4xSDL-rw

In our opinion, this level of performance (~4 bits/sec) unlocks many occupational use cases for mouse use, and while more bits/sec is always better, some users may start to prefer additional functionality (e.g., right clicking, scrolling, keyboard shortcuts) over improvements in point-and-click bitrate.

How does the MouthPad compare to point-and-click brain computer interfaces? For the 44 hand-impaired head control users and 23 typically-abled head control users, we report and compare max performance as well as “typical” performance, as both are important. Maximum performance demonstrates what the interface is capable of. Typical performance reflects what a user should expect when they purchase a device. Specifically in the case of the MouthPad, this is typical initial performance, as most users perform the assessment only once as part of onboarding.

The maximum performance of the highest scoring MouthPad user exceeds any of the other interfaces. 45.45% of MouthPad head control users achieve a higher BPS than the best maximum point and click performance achieved by the best published point-and-click BCI [2], which notably excludes Neuralink’s results as they are not peer-reviewed. 15.91% of MouthPad users score higher than 7 BPS, which is a level where we subjectively feel is past the sufficiency threshold for casual and professional computer use.

What about other brain computer interface outputs? Many BCI researchers have moved on from cursor control to other future-looking technologies like decoding speech [8, 30, 31], handwriting [32], or finger movements (e.g., for keyboard touch typing) [33]. As language outputs, the words/characters per minute reported by those studies could be compared in terms of bits per second. But doing so requires knowing the entropy of the words used in decoding - clicking random targets in a grid is a reasonably realistic simulation of some computer tasks, but generating random sequences of letters or words is not. Typically participants in those studies are using BCI for language output because of difficulties producing intelligible speech. While decoding performance is very impressive and encouraging for that use case, the output rate typically falls short of spoken language speed. In the case of the MouthPad, users can nearly all speak - perhaps with some volume impairment or altered phonation, but typically intelligible speech. Rather than trying to match those impressive language BCIs with the MouthPad, we instead designed VOX to meet our users where they are and translate quiet, body-conducted speech into text.

How does the MouthPad compare to other assistive technologies? To make this comparison, we must make a slight detour into the details of the performance metrics. Typically published results in alternative pointers has quantified performance using a Fitts’ law measure of bits per second.