It is very normal/usual/expected that a GPU won’t run at 100% unless you provide it with enough parallel computations to perform (either extremely large layers or a big batch of samples). There’s just so much overhead in so many places, it’s impossible to be efficient at a small scale, and there’s no one thing we can point to and say “that’s why it’s slow”. My best advice is to look into the optimization possibilities provided by the model/framework/version you’re using. I’m in pytorch, and use things like torch.jit.script, torch.jit.trace, torch.compile, torch.autocast, etc.

I think it depends on whether you want to go into a more researchy or more practical MLE role. I did the capstone project track, and don’t regret it as it let me pursue a larger variety of projects, and I could also do an independent study that ended up being kinda Thesis Lite™. If you do a full thesis then the majority of your time in grad school will be spent on just one project, and a lot more academic research than practical engineering. It will definitely set you up better for doing more cutting edge stuff in the industry, which could potentially be more fun/interesting, but if you try to take your career in that direction then you’ll be competing against a lot of people with PhDs.

abs(x)?