Eye Gaze Tracking With Head Tracking Solutions On Linux

Question

Background:

I've been dealing with chronic RSI for years and I'm looking into various options to control the computer without my hands. Linux has been my OS for a few years and I really don't want to go back to Windows if I can help it but if I can't find anything very soon I'll have to count my chips because I've already spent many hours looking.

The Meat:

In particular, I'm looking for eye gaze tracking with head tracking (preferably in the same program) but I would entertain hand gesture technology as a complementary program. Any price is fine, but free and open source is always nice. If you have any recommended hardware for tracking please offer your suggestions. Because the eye naturally jitters (or saccades) when you look at a fixed point, the advantages of this technology is the speed of eye gaze tracking with head tracking for single-pixel precision. I would be using this along with a voice recognition stack to control my computer and even code.

I found Precision Gaze Mouse that works with Eviacam but unfortunately Precison Gaze Mouse only works on Windows at this time. I'm looking for something like this combination. The Precision Gaze Mouse video at the link below demonstrates how it works. Here are the links for reference:

https://precisiongazemouse.org/

https://eviacam.crea-si.com/

Your question might be better suited for Software Recommendations. — Eduardo Trápani, Dec 05 '21 at 19:18
I agree but Software Recommendations has about 1/7th the user base of the Unix & Linux community. It's a fairly difficult question and time sensitive so I thought my chances would be better here. I may try that if I don't get any suggestions soon. Thank you for your input. — Growing My Roots, Dec 06 '21 at 23:50

score 3 · Answer 1 · answered May 03 '22 at 11:30

Don't get your hopes up, but hello hope you're good. I wanted to make a free software something like this for Tobii eye trackers on Linux (Tobii 4C and Tobii 5). I also hoped to put it on a Raspberry Pi, and have the Raspberry Pi pretend to be a mouse (Linux gadget, USB On-The-Go). But after many emails they said I could only make it proprietary and I would need to buy a license because they're only targeting WinDoWS gAmERs, and I gave up.

There is a thing called Talon, which has Tobii support: https://talonvoice.com I've not used it, it's a big bloat proprietary package, but it's something.

I had set my hopes on Tobii because they were affordable, most gaze trackers were for academic purposes. Here's a comparison: https://thume.ca/2016/03/24/eye-tracker-reviews-pupil-labs-tobii-eyex-eye-tribe-tobii-x2-30

I didn't find any good gaze tracking implementations using a webcam, but I feel it could be done. The best I found was this: https://github.com/antoinelame/GazeTracking But it's more just eye tracking and "looking left" or "looking right", no gaze to point on screen projection.

I use my voice instead of a keyboard using my own setup: https://git.sr.ht/~geb/numen I don't use a mouse, I just use keyboard-centric programs like the Vim text editor and the qutebrowser web browser. The plan is to get it working on a Raspberry Pi so you can use any computer by plugging in the Pi. I'd just like gaze tracking for things that require a pointer device like drawing diagrams.

Maybe something I've said here will be helpful.

From my research on web cams it seemed that Tobii was one of the better options as well. I've heard of talon but haven't tried it. I might if I get a little more desperate. That'd be great if you could get some thing working on a pi, I might try your implementation at some point. qutebrowser looks pretty cool too. Appreciate all the information! By the way, I came across a way to access the Vosk–API via nerd–dictation which may help you in your project. This post includes a demo video https://unix.stackexchange.com/questions/256138/is-there-any-decent-speech-recognition-software-for-linux — Growing My Roots, May 03 '22 at 22:34

score 2 · Answer 2 · answered Jul 23 '22 at 21:04

I'm also with RSI and I'm using talon with a tobii 4c since march 2022. It's a free batteries-included package with a growing community. In my case I was very happy about it, since its setup is very fast and I could use my linux system with it again. I'm still tuning my setup and there are definitely things I don't like about it. But in my opinion its user-experience (including python scripting) is over the average and it's by far the best start I know of. You can also use talon only as an interface to the tobii hardware and nothing more:

# this is beta-API, the free version differs a tiny bit in its naming; leave a comment if you're interested
import pickle
from talon.plugins.eye_mouse_2 import BaseControlMouse
class StreamingMouse(BaseControlMouse):
    def update(self, tracker, screen, frame):
        # ~70 Hz on my system
        socket.send(
            pickle.dumps(
                (
                    frame.gaze,
                    frame.head,
                    frame.gaze_frame.ts,
                    frame.gaze_frame.left.gaze.x,
                    frame.gaze_frame.left.gaze.y,
                    frame.gaze_frame.right.gaze.x,
                    frame.gaze_frame.right.gaze.y,
                )
            )
        )
streaming_mouse = StreamingMouse()
streaming_mouse.start()

https://github.com/adabru/speech/blob/main/adabru_talon/code/eyeput.py

In the long run I want to replace it with open source solutions because of some practical limitations. I'm thinking of Vosk and Pupil Labs (though I'm more interested in the VR version). — adabru, Jul 23 '22 at 21:05
Thank you for sharing your experience and code. This may prove to be helpful. What are you using for speech recognition, if any? — Growing My Roots, Jul 26 '22 at 20:17
I'm using talon's builtin speech recognition. The neural network is trained by talon's developer. — adabru, Jul 27 '22 at 21:41

Eye Gaze Tracking With Head Tracking Solutions On Linux

2 Answers2

Linked