I realized not too long ago that I hadn’t fully told the story of how I created my own “Jarvis” assistant using Claude and a little bit of programming know-how. I’ve had this idea for years to have an earpiece in your ear that could contact a server that runs an AI assistant running on a custom architecture.
Ten years ago this was something that was light years ahead of my technical chops and really a pipe dream. Something Tony Stark would have created. I eventually gave up on the idea, convinced it would never become a reality. But the dream never really died.
So one day I innocently asked Claude: How much work would it take to create a “Jarvis” in your ear earpiece that connects to a home server and acts as a web search assistant using voice commands? Claude quickly assured me it could be accomplished in a weekend using Python and modern tools, and then asked if I would like him to write some starter code? I apprehensively typed yes.
And so Claude and I went through several iterations of the code and I had a working program that night before my usage limits ran out. I was absolutely blown away. It talked and ran a local model on my laptop that cost me nothing except my time and was able to answer all sorts of questions with amazingly complex answers.
I quickly realized I would be spending a lot of time on this project.
I learned along the way that Claude does occasionally make mistakes but is actually right more often than I am. I also learned a lot from Claude about software architecture and how the code is structured on the backend by looking at the code to verify what it was telling me. It does help to be able to read Python code yourself.
One mistake I made was allowing the natural language processing for the text-to-speech to be run in the cloud via ElevenLabs but I now have a local solution through Piper that is very good and keeps the whole project local on my machine.
I also learned a lot about console program design – more than I ever did by writing dense C++ code. One major overhaul I made to the program was by refactoring the project to use a more minimal interface that copied the interface of Ollama’s console app in some ways. This interface design made a real difference in the enjoyability of using the program.
I found using a free Mistral model via Ollama was really critical to making sure the project was accessible to me as a programmer and kept a lot of the design decisions abstracted behind the Ollama API (though AI kind of levels the playing field when it comes to writing efficient backends). I now use the Gemma 4 model by Google but that’s mostly a personal design decision.
Something like 70 iterations and several weeks later (because I was only using the free version of Claude, and had to wait till the next day when my credits would reset) I had renamed the project to Friday after the character from the book Robinson Crusoe by Daniel Defoe (Though the first thought that ran through my mind was of Tony Stark’s Friday assistant).
It talks with an Irish accent and though it takes a full minute or more to respond because it’s just running on my laptop, it really is impressive and shows what can be done with minimal experience by a technically inclined person.
I do think that if you want to learn to be a software developer you need to learn to rise above vibe-coding in a way. You’ll never learn the hard skills of syntax and program structure that way (not to mention hardware concepts).
But there is something to be said here for someone who already has those chops and just wants to create something amazing quickly and easily. With capable AI agents, software development is truly not in the realm of the obtuse and deeply technical anymore.
If you want to see the code behind my Friday/Jarvis project or run it yourself, you can see it at my Github repo.
Timothy Grindall is a technology writer and self-taught computer programmer, photographer, filmmaker, thinker, and writer who wants to see the world change for the better.
Leave a Reply