The Product Vision

Since I didn’t have much time to spend on this project I used it as an opportunity to experiment with LLMs for embedded coding. Claude and Codex have been great for whipping up React prototypes in Typescript, but Circuit Python on embedded hardware is a very different challenge. While the syntax is easy (it’s just regular python with different libraries) the hard part has always been figuring out what hardware you have, finding the right drivers, and dealing with APIs that have changed over time. I’ll come back to how well it did, but first let’s get into the product design.

My vision for the device was pretty simple. I want a clock with an option to say On Air when I’m in a call. It needs to be big and bright enough for someone to see from across the room. While every good device should have a few buttons or even better a rotary encoder, anything more complex needs a display. I originally planned to have the setup on the device but I quickly learned that making even a simple menu with text is pretty much impossible on a 16x16 pixel display. Since this board has WiFi I figured it could run a web-server pretty easily, letting me push the complexity into a web app. This proved to be a wise choice. Not only can LLMs quickly spit out pretty HTML interfaces, it meant I could build a REST API that would be useful later for integrating with other tools.

Initial Prototype

For the hardware I began by digging for random components in my parts bin: a flexible 16×16 RGB matrix i bought some years ago on a whim and never did anything with, a random ESP32 based board I’d gotten from Adafruit, and some spare USB cables.

For the initial software I asked the LLM (I can’t remember it was Codex or Claude) to connect to the board over a serial port and print hello world on the display. This didn’t work because I had my pins wrong, but the LLM was able to walk me through debugging it. Once we could reliably control the pixels progress was quite fast.

My vibe coding style is lots of short prompts with fast iteration. Together we quickly got a clock, timer, and scrolling Quiet Please text. That part was fast. Then I spent twice as long futzing with colors and animations until I had it how I wanted. The agent was able to generate placeholder images for graphics but tiny pixel art is not an LLM strong suit, so I later drew replacement PNGs by hand.

Once we got the web-server up and running I realized how powerful this development loop was. I could easily experiment with different ideas, many times faster than I could by hand. I could ask for a feature, let it spin for a few seconds, then reload the webpage on my phone. Most importantly, the latency of a REST call from the web app to seeing the screen change is essentially instant. I haven’t measured the exact latency (which would require using the high speed camera on the phone I was using for testing), but the lag was low enough that I could use a tiny live pixel editor to draw artwork on the actual device.

Making It Pretty

Once I had the basics working I focused on making it nicer; something a non-technical user would enjoy, without the wires and LEDs splayed everywhere. I experimented with sheets of paper as a diffuser, confirming that the text would be far more readable with something in front of the LEDs, even if it didn’t have full proper boxes for each pixel.

I started by 3d printing a stand with a curved groove to hold a sheet of paper as a diffuser and it worked well. Of course this early success meant I wouldn’t be happy with just a slip of paper. I needed the diffuser itself to be 3d printed. Mwaa haa haa.

To make a curved grid diffuser, I watched a few YouTube tutorials to learn how to make curved surfaces and printed a model plain white PLA. I polished up the web app and got MDNS working so anyone can load ledclock.local in their browser, and if they are on the local wifi it works, loading almost instantly.

After a few rounds of polish this is what I had.

I’m really happy with how this turned out. Now let’s discuss what I learned.

Agentic Embedded Development

An agent can do well when it has the ability to validate its work. It can put code into the board and parse the serial output to see if the code does what it expected. This lets the agent quickly diagnose missing dependencies, bad pin assignments, and even find rendering bugs, though I have to be the agent’s eyes to report whether something worked visually. If the agent had a camera pointed at the led matrix it could probably run autonomously for quite a while. That’s something of like to try in the future. (Until it leads to the robot uprising).

Local Web Apps are powerful

I can’t stress enough how good the experience is of using a web app on the local network with MDNS. I can tell everyone in the house, even my nearly 80 year old father, to just type ledclock.local into their browser and then they are instantly connected. Low latency REST calls make it such a nice experience. I didn’t need to spend any time optimizing for network latency, and instead spent all of my energies on building an inituitive user interface.

The CircuitPython MDNS API is easy to use and can be configured by an INI file on the root filesystem of the device. Whenever I need to change any of the network settings I can edit the settings.toml file on my laptop in any text editor.

Next Steps

The biggest flaw right now is that it needs two power supplies. The LED matrix has a power source separate from the MCU. In the future I want to find a USB-C powered board that can also supply the higher current that the display needs. I also want to clean up the 3D printed enclosure with smoother edges and a single rotary encoder for quick interactions (pause timer, toggle quiet mode, etc).

Given how easy it is to create embedded hardware products with Circuit Python and agentic coding, why aren’t we seeing more small run devices that don’t lock us into cloud subscriptions. Maybe it’s time for some new Kickstarters.

As always, you can find the code in my github repo.

The Dawn of Smartphones

Let’s go back to 2005. The height of mobile technology is RIM’s Blackberry and Palm’s Treo smartphones. Cell networks are getting better and Microsoft is threatening to turn Windows CE into something people would use without being forced to by their employer. And all the while rumors are swirling about the iPod of phones that Apple is cooking up. In walks Google.

Google’s dominance in search is becoming a certainty but their senior management has found an existential risk: fundamentally Google is a website. Someone could eventually make a better search engine. Even worse, the owners of the operating systems control who the default search engine is. Google can pay for that privilege (which they do), but they know it could be taken away at any moment. They could always be intermediated by the platform owner.

If you are Google in 2005 you know the only way to never be cut out is to own the platform the end customers use to get to you. So they formulate plans to replace every platform that could hold them back. The web browser is a platform, so they start planning their own web browser (Chrome). The desktop operating system is a platform, so they start planning ChromeOS. Then they look further afield.

The future is clearly mobile, and if Windows Mobile owns most of the market like MS Windows does on the desktop, they could be cut out as well. The solution: buy your own mobile operating system and get it everywhere before the competition reacts. And it worked. Android is eventually on more than 2/3rds of the worlds phones and Google is on almost all of those. Paying off Apple takes care of the rest. World Domination Accomplished. Well done, let’s knock off early.

Cut to a few years later. You are Mark Zuckerberg. You want an empire like the big guys. Your little social website is doing pretty well so you burn cash trying to build or buy any possible threat. You retool your website to become mobile first. You buy What’s App to dominate messaging. You buy Instagram to dominate photo sharing. The problem is: you still don’t have your own hardware platform. You could be banned from the App Store at any minute. That’s an existential threat. What can you do?

Facebook Makes Hardware

Facebook made a phone. Sure it was just reskinned Android and manufactured by HTC, but everyone called it "The Facebook Phone". It came with Facebook Home, a UI layer that let you Facebook your Facebook without even starting Facebook. Sadly for Mark, it was a flop. It turns out smartphones are a rich man’s game and even Facebook wasn’t rich enough to win it.

Mark is a pretty determined guy, so he invests in the Portal, their attempt to put video calling on tablets and TVs. (I forget if they changed their name to Meta by the time this shipped). I actually liked the Portal TV device; with some UI work and an open platform I think it would have been successful, but it simply didn’t do enough to earn its keep on HDMI 1.

Clearly hardware is hard so you need to buy your way in. Facebook bought Oculus, makers of the wildly successful Kickstarter backed VR headset, focused on making VR cheaper and more immersive thanks to the advent of cheap smartphone sensors. For a while they let Oculus run as a separate (but very well funded) group, but eventually it was sucked into the mothership. I actually interviewed to join the Oculus group a few weeks before the great sucking. Dodged a bullet.

Facebook now has a successful hardware platform where they control the entire stack. While it was built on top of Android instead of building from scratch. (Thank you, John Carmack), the Quest had its own shell and its own browser (no Chrome here!). They were in charge of their own destiny. Smooth sailing, right? Time to unify the disparate parts into a 100% Facebook virtual experience that you can’t get out of: Facebook Horizon

VR was dying before COVID

Let’s get to one of the big points of the article I started this rant with, that if VR couldn’t be popular during a global lockdown when everyone had lots of free time and was trapped inside, then it would never be popular. While that might be true for Horizon, it is not true about VR in general. Truthfully VR was dying before COVID. I know because I was there.

I worked in Mozilla’s Mixed Reality group from 2017 to 2020. I saw all of this first hand. Most investments were turning sour before COVID.

• Google launched Cardboard and later Daydream, platforms that turn your phone into a VR headset. They did a full marketing launch (read expensive) with hardware partners like Lenovo in 2016. It was discontinued in the fall of 2019.

• Microsoft launched Windows MR in the fall of 2016 with partners like HP, Dell, and Samsung. They didn’t sell well (and were very slow, as I recall), and they started dialing back the marketing before COVID hit. Microsoft formally laid off the teams in 2023.

• HTC, an early VR pioneer with the HTC Vive, pivoted to enterprise use in 2020, as did Microsoft with the Hololens 2.

• Sony announced a VR headset for the PlayStation 4 in 2014, and released in 2016. Few games were released for it and I’m honestly surprised they shipped a second version of the hardware. The PlayStation 5 does not support it other than with older games using special adapters.

• Pico Interactive shipped an early smartphone based headset in 2016 and its own standalone headset, the Pico 2, in 2020. I quite liked this one (it ran Firefox XR as the browser), but it was the beginning of their pivot to enterprise development.

While some of the shutdown announcements happened after COVID began, the writing was on the wall before the virus hit. People just didn’t like VR that much. There wasn’t much to do. A few games. Some 3D videos. Some chat rooms. To the average person this was just the mobile experience with extra friction.

The Bubble Popped

The big investment in VR started around 2015 and began to peter out just as the pandemic hit. If anything COVID stretched VR's life, not cut it short.

Certainly Facebook sucked all of the air out of the room by selling their hardware at a loss, making it impossible for other startups to compete in the VR hardware race. When the other big companies pulled out — mostly before COVID -- Facebook had the playing field all to itself and a pandemic to make everyone stay home. So what happened?

Fundamentally there was a VR bubble and it popped. The companies pouring billions of dollars into the industry realized that it was a useful technology and would sell some hardware, but VR would never become “the next smartphone”. It would never be as big as the hype claimed and the revenue projections demanded. It would never be a big enough consumer market to be interesting to what are some of the biggest companies in the world. So the bubble popped, which I also expect to happen with the AI bubble in the next 12 months.

At the end of the day the VR ecosystem died because it wasn’t providing compelling experiences that people cared about. This wasn’t Facebook’s fault, but they didn’t fix it either. They did the opposite, in fact. They built Horizon OS, a closed platform. Let’s come back to why Facebook wanted VR in the first place: to solve a problem.

A solution to a problem only Facebook has

Successful products solve a problem. The Metaverse, as conceived by Zuckerberg, was not a solution to a problem that regular people have. It was pitched as a way to immersively bring people together. That’s not a problem most people need to solve. And it wasn’t even the real problem the metaverse set out to solve. The metaverse is a solution to a problem that only Facebook has, and that problem is the fact that Facebook doesn’t own a platform.

Mark Zuckerberg has long wanted a hardware platform to call his own. A place where he can call the shots and not be intermediated by anyone else. Even better if it has eye trackers built in. ( He really, really wants this). I agree it’s an interesting problem, but it’s not a problem that regular people have. It’s a problem that only Facebook has. That’s why Quest has been a failure for Facebook. That’s why no one bought the Facebook phone. That’s why the Portal was wasted potential. If you want people to join your platform then you have to make it be a solution to a problem that people actually have. Facebook didn’t do that. They should have looked towards the web.

The Open Web

The Metaverse failed where the Web succeeded for a simple reason: the Web has always had permissionless innovation. It has never been owned by anyone. Anybody can set up a website. Anybody can make a webbrowser. A thousand flowers bloomed a thousand times over. There were plenty of big companies trying to make something like the Web “happen” in the days of dial-up. And they all lost, because the Web was open.

The Metaverse was never open. It was conceived by Facebook as something that Facebook controlled. It could have been like the web, with people jumping through portals as we follow links today; but that wouldn’t have given Facebook control. Permissionless innovation was a threat to them. The irony is that the permissionless innovation of the web is what let Facebook become the social media giant it is today.

Facebook actively fought against there being a VR ecosystem. They sold the Quest at a loss so no other hardware maker could afford to be in the consumer market and iterate on their hardware. Once they had the market all to themselves it didn’t grow any faster than Facebook’s own software ecosystem; which they immediately slowed down. When an indie game became popular Facebook bought them. That’s great for the particular game, but showed to the rest of the game market that Facebook would Sherlock you in an instant. So no one else would invest.

Next Facebook spent untold billions making a VR social network that no one wanted because there was nothing to do there. They built instead of growing. The Web has graceful degradation so that older hardware is included. Facebook made their VR OS slower over time, to the point that first generation Oculus devices are unusable now. Thank you, Carmack for letting us root our Oculus Go’s.

Facebook could have made it easy to launch a VR experience in your Quest with simple app integration. When you find a cool webpage, try it out briefly in the desktop browser, then click "Share to my Quest" and pop on your headset to try it out in full fidelity. But they didn’t do that (not well, anyway).

The good stuff that builds a new platform doesn’t come from big brands and partnerships. It comes from the little guys experimenting. YouTube didn’t start with Hollywood content. The early success stories were people making silly stuf fin their spare time. New platforms have to grow organically. You can’t just throw a bunch of big brands and marketing dollars into a blender and spit it out. That’s how you get Quibi.

Facebook could have made it super easy for people to create WebXR experiences with great dev tools, public domain models, and open source toolkits; but they didn’t. The permissionless web was a threat to them. Building a successful platforms implies that people on the platform might make money that could have been yours, and Facebook can’t give up that control; so they got to have their own little clubhouse and threw a big party... and no one came.

Now they are trying again with AI and XR glasses. Good luck with that.

Coda

I don’t want to blame the death of VR on Mark Zuckerberg. Some of this is just typical big company behavior. Big companies get bored and can’t have long term visions. If something seems like it will be profitable but not wildly profitable, then they abandon it. Occasionally big tech will keep it on life support to meet the terms of contracts they signed, but their heart isn’t in it anymore. How on earth did the Kin launch when everyone involved knew it would be killed immediately?

Throughout this piece I haven’t mentioned Apple. While Apple did release a headset in 2024, I was never meant as a consumer product. The Apple Vision Pro was meant to test the waters, collect data, and built up their developer platform before the real AR product (probably some glasses), ships in the next few years.

I will say this about Apple; for how much I complain about their aggressiveness in shipping new features at the expense of software quality, Apple can sustain long term vision. They can plan and execute and iterate until the third one is good. Google can’t do that. Microsoft can’t do that (anymore). And Fadebook can’t either.

Vibe Coding

For the past few weeks I’ve done some "vibe coding" with Codex and Claude in my free time. As a lifelong software engineer my feelings on the results are, well, mixed.

LLMs definitely let me put together a prototype faster, which is great for iterating on ideas quickly. The generated code is definitely an improvement over state of the art a year ago; generally straight forward if rather verbose. For UI work I need to hand hold the tools, but that’s probably okay. I’d do the same with a junior engineer. What I’m contributing is my experience designing interfaces. I can tell the LLM to “move this button over here”, “make this label bigger”, and “fix the scrolling of this table on that side”. These are the same things I’d say to a junior engineer and I get the results much faster; definitely a win in these areas. And I’ve a blast working on projects that have been on the back burner for years. Whipping up my personal cloud based iTunes in a matter of hours. I won’t deny it’s been fun. But...

Aye there’s the rub. I can only be effective at vibe coding because I have these decades of experience. The LLM can easily code itself into a hole. Last week I went through a few rounds of the AI fixing horizontal scrolling by breaking vertical scrolling, then fixing the vertical and breaking horizontal again. After 30 minutes of in this loop I had to fix it by manually rewriting the markup to properly use CSS Grids. To be fair, plenty of human engineers get stuck in this loop as well. Nested CSS scrolling is tricky and LLMs aren’t magic. They need experienced guidance to be effective. So who can provide that guidance?

The Decline of Software Quality

I am 50 now and wondering where my career goes next. For a while I was worried that AI would cut me short and force an early retirement. Now I realize AI tools will have the opposite effect. Engineers who know software at a deep level can coax these tools to do a better job. Now I worry about the juniors. If newcomers never get the experience of building software by hand then this will be the last generation of engineers with the ability to produce quality software. In the future we will prompt and get the results from AI but software itself will never get better. And if AI is trained on the code of today, then it will never get better either.

I also worry about the long term quality and viability of my industry, the commercial software industry. With AI slop a solo software developer can ship faster but never sell a single copy because their products get lost in the noise of the AI slop marketplace. This is not new, however. Video games have been taken over by slot machine mechanics. Free to play. Consumable in-app purchases. Harpoon the “whales” and let the rest of the players rot. Small useful utilities are slathered ads and tracking. Want to generate a QR code from a URL or make your own LOL cat meme image? Better come armed with the latest in popup blockers and antivirus tools.

Software on the Internet, which is virtually all software these days, has become completely user hostile. Even if it doesn't steal your data and shove ads in your face it is still ruled by algorithms which optimize for "engagement", and value rage bait over thinking and conversing.
I didn’t get into this industry to make such crap.

Somewhere around the time of the pandemic I started to withdraw. I let my social media accounts atrophy and recently turned them private. I started focusing more on family and local issues rather than trying to change the world. I got into Computer Science as a wide-eyed youth because I saw the power of software to help people. To make the world a better place. That doesn't happen anymore. I have seen little come out of software in the past decade that meets the goal of improving the world. I would be perfectly happy with my iPhone 6, an old Macbook, and the Internet of the early 2010s.

Perhaps this was always the end state of my industry. In many or possibly even most cases, the quality of software does not matter. I’ve worked for many different startups and many Fortune 500 companies over my 30 years in the industry. Almost never did the success of the company depend on the quality of the software we produced. On the contrary, software has become worse. It has become disposable and user hostile. In a world where profits come from engagement the software must be exactly good enough to not drive users away too quickly, and not a whit more.

AI slop accelerates this trend but was not the cause of it. Now software can be made even cheaper and faster and of ever lower quality. The flat-pak fiberboard furniture of computation. Efficiency doesn’t matter if the hardware gets faster and the goal is just to suck as much money as possible out of people’s wallets before they throw the app away.

Software That Sparks Joy

The enshitification of software is a long term trend. My choices and actions will not change that trend. However, I can choose how I respond to it. So this is what I’m going to do.

I like my day job working on financial software. It’s one of the few areas where quality really does matter because bad financial software that loses customer funds doesn’t last very long. In all of my other projects I am going to focus on one thing: software that sparks joy.

Many see software as disposable. And perhaps it is. But I still want to make quality software that brings concrete happiness and value to end users. Not to the giant software platforms. Not to social media. Not to the latest AI VC backed company. Software made by me, a human, for someone else, another human, to use.

About 20 years ago I found an MacOS application that lets you build a comic book out of your photos called Comic Life (hat tip to reader Daniel W for reminding me of the name). My niece and nephew enjoyed designing and printing their little adventures with speech bubbles and halftone image filters. This is software that sparked joy, even if only for a few people. I want to make software like that again.

Will I use AI to help me write my software? In some areas, sure. I wouldn’t build furniture without power tools. I’m not going to put together a React CRUD app without using the tools that let me do it the fastest; but I’m also not using these tools to make worse software. I want to use AI to free up my human attention on the parts that matter the most. The user experience. The efficiency. The quality. The innovation. The artistic intention. That’s where (I hope) I can make a difference.

I have built a few free webapps like this. Box Builder makes a box and lid from your specifications, immediately downloaded as a 3D printable STL file.
The TextEffect generator does what it says on the tin, generates a text effect and gives you the CSS and downloadable PNG of the glyphs. Meem Makr is my first vibe coded app. Drag your cat photo in, get a funny lolcat out.

I'm putting all of these on apps.josh.earth until I figure out a better place for them.

Am I wrong? Am I just a grumpy old man yelling at the kids to get off my carefully coded lawn?
Does it even matter? This is what I do. I make software. And I want to make it for humans because I care about humans. Sometimes, they are kinda neat.

Greenfield

These tools work better on new code than existing code bases. I can describe a new application in prose and get a pretty good prototype. The tool I used, Codex, significantly struggled to fix overscroll issues with the web-based music player I'm working on. To be fair, plenty of human coders struggle with this as well. When I build such layouts by hand I have to plan ahead and stick to strict rules. I'm not surprised the AI failed as well.

Closed Form

It is good at things with a well defined answer. If something can be checked by a unit test then Codex can easily write and verify it. UI is harder because it's open ended and requires a lot of intuition, so it does far less well at.

Product Management

Working with Codex essentially turned me into a product manager. I would ask for something, check the results, make suggestions, and repeat. I was able to build a web -based itunes like app very quickly, but I spent hours on the little details. At some point it would have become faster for me to make the changes myself. That said, describing exactly what I wanted in prose forced me to really think throiugh my goals and be explicit. I would not have had to describe to a human programmer (who had some UI building experience) that "When jumping to an artist by typing in the name the list should scroll so that the newly selected item is in the viewport.", but I did with Codex. I have to carefully use the UI to find issues. It was refereshing, however, to be able to act as just a product manager and not also the programmer when working on personal projects (as this experiment is).

I think that AI coders, at least for UI work, care free up the human from drugery, but still require close monitoring. I definitely worry about the quality of the generated code. It's not bad, but it is very rote and can be hard to follow. It looks like what I'd expect of a junior engineer.

Outside In

Outside In is where the parent always does its own layout first, then it provides available space to its children. This is typically good for embedded where parents take up all available space and divide it among their children, but it does rule out certain layouts. What if a row panel wants to be as wide as it’s parent but only as tall as needed for its own children? Then it needs to ask the children for their heights first. Outside In won’t work in this case.

Inside Out

Inside Out means the children choose their own size based on their contents and the parent only positions them. A parent can shrink itself to fit the size of the children (as in the row example above), but children cannot expand to fill all of the parent’s space. However, this does result in a very fast and deterministic system. We only need to traverse the tree once and parents always have a pre-sized children.

Two Phase Layout

The parent passes available space to the children and the child sizes itself. This lets a child expand itself to fill the available space if it wants to. But how does the parent know how much space it has available? It gets it from its parent, like in the Outside In layout. But it also might shrink the available size before handing it to its children. This requires two phases of layout. The children’s minimum sizes are measured, then parent calculates how to distribute the space, then the children size themselves using that space. This is the classic layout in many UI toolkits but results in an exponential cost algorithm. At each layer we are doubling the number of calls. If a child needs to look at its own children to determine it's minimum size then we have at least an O(N^2) algorithm. Not good.

Fixed Layout

Simply have the developer manually size and position everything. This is actually the fastest. Now our layout algorithm is O(1)!. It's actually pretty common on embedded systems since there usually are resizable windows. However, it requires the developer to do more work and doesn’t handle cases where the content itself changes, which is likely with any Internet connected device. Imagine a button that doesn’t become bigger if the text is longer? We can do better.

Flutter

I really don’t like any of these solutions.

Flutter’s algorithm is pretty good and sub-linear in many cases. Flutter has the parent provide constraints and the child returns its preferred size. For something like a vertical box with flexible children the parent must lay out the non-flex children first to find the leftover size and then lay out the flex children. In order to remain non-exponential we need a way for the children to indicate they are flex without actually calling the child’s layout method. Flutter handles this by actually doing two passes and agressively caching results so that the common case is sub-linear even if the worst case is exponential.

The Compromise

Unfortunately, we can’t use Flutter’s system directly. It was optimized by many talented engineers over many years. It also uses extra memory for the caching. We don’t really have that option for a tiny open source library targeting embedded. So I came up with a compromise. Instead of calling a child method to get its preferred behavior we the child has flags for flex and alignment. The parent can check these flags without invoking an entire layout pass on the child. This gives us most (but not all) of the behavior of Flutter’s system without caching and being reasonably simple to use. There are some cases it doesn’t handle gracefully, but it works well for the most common ones so I think this is a reasonable tradeoff.

The New Layout Proposal

So here is how the new system works.

• A view has h_flex and v_flex attributes set to Flex::Intrinsic or Flex::Resize.
• a view has h_align and v_align attributes set to Align::Start, Align::Center, or Align::End.
• These flags are set before layout begins.
• Children set their size but their parents set their position.

Layout processed as a single pass tree traversal. A parent:

• izes itself based on available space passed into it.
• for all Intrinsic flex children, calls child.layout(space) to have the child determine it’s size.
• calculates the remaining space and divides it by the number of Resize flex children.
• for all Resize flex children, calls child.layout(remaining) to have the child determine it’s size.
• set the position of all children using their alignment.

For complex parents like vertical and horizontal flex boxes they will perform all the steps above. For simple views like buttons and labels they will set their flex to Intrinsic and calculate their size based on the font and title text.

So far (barring unfixed bugs) the above system handles everything I’ve thrown at it and works quickly with no exponential layout explosion. It even worked for a simplified CSS grid layout where the developer can define rows and columns and assign children to different cells with different alignments.

Next Steps

Now that the new layout system is merged from a branch I’m getting close to a 0.1 release. I still need to:

• Document the event loop.
• Document how to create a custom view.
• Pick a better project name.

If you have any questions for feature requests, please get in touch: josh@josh.earth.

Project Repo

Goals

The goal for this embedded GUI lib is to be able to quickly create basic and fast UIs on any device that supports Rust’s embedded_graphics API. Since most of these devices put the display on the SPI bus, they tend to be fill-rate limited. This means it’s very important that the GUI architecture lend itself to avoiding overdraw. The lib does this by tracking dirty rectangles and using a clip rect to only draw what has actually changed on screen. It also means the screen is only redrawn when something actually changes.

View Graphs in Rust

GUIs are traditionally done using a tree structure, however this is something that is especially difficult to model in Rust due to the borrow checker. Most GUIs end up becoming a spider web of references, which Rust won’t let you do. To address this challenge I’ve come up with a few stratgies.

• No subclassing. Inheritance is simply a bad fit for Rust. We can sort of do it using traits, but it’s messy and error prone. Instead we have a single View struct that defines all views. It contains the basic state that all views need like a name, bounds, if it’s visible, etc. To create a Button or Label, we instead of using subclassing we set optional fucntions on the struct, sort of like methods, which leads us to:
• Function references for customization. If you want to create a button then start with a view and make a draw_button fuction and set view.draw = Some(draw_button) to use it. The same pattern is used for state, handling input, and doing view layout. If a view doesn’t need that functionality then it sets that attribute to None.
• View state is moduled using a struct that is only available to that view. Input on the view is only used for modifying that view state. Anything which affects another part of the application happens externally. For example, a toggle button has an internal state struct with a boolean indicating if it is selected. On input the toggle button can modify that itnernal state. Something in the app which needs to change when the button is toggled would not customize the button. Instead it would respond to an action event triggered by the button.
• Responding to input happens outside of the View. Manipulating application state should not happen in the views. The views only modify their own state using input. To modify other state the application should listen to raw events on the scene or handle actions generated by views which are then returned to the application. This separation means Views are entirely isolated. They care only about their internal state and are easy to test. We never add an event handler to a view. Instead we trigger handlers when views produce actions.
• Event handling goes in the main loop. A typical app has a loop to process input events from hardware (keyboard keys, touch screen touches, etc) and then redraw the screen if anything in the app has changed. This where application logic triggered by events should happen, not in the views.
• Because parent child relationships are hard to modle using trees in Rust, we don’t use a tree. Instead all views are owned by a Scene object which tracks the parent child relationships. It provides utility methods to add a child view to a parent, get a list of all of a parent’s children, and draw the views in the correct order. Views are always reference by Strings rather than pointers to the views directly, avoiding ownership issues. Under the hood Scene uses a hashmap of vectors to model the logical View tree.

Progress So Far

The Scene and View work pretty well. There are built in components for button, label, text input, toggle button, toggle group, panel, and a popup menu. All components are generic. They draw using the DrawingContext and Theme abstractions which are generic over Color and Font. This means a View can be created that will work with any drawing surface and any theme with any color and font type. The views draw using semantic colors and fonts provided by the Theme at runtime. I'm not completely happy with the type signatures. There's so many generics that you have to copy everywhere, but it seems to work.

To use this library with a real device you need to provide implementations of DrawingContext and build your own app loop which generates input events and redraws the scene. Look at the esp-test example to see how to do it.

To create your own views just assemble one at runtime. Build a View struct with your own values for state, input, layout, and draw. To have your code be notified when a view receives input, make the input function return an Action enum with some indication of what it did. Ex: a list view could return Action::Command(item) indicating which item was selected. Here's an example of a toggle button:

pub struct SelectedState {
    pub selected: bool,
}
pub fn make_toggle_button<C, F>(name: &str, title: &str) -> View<C, F> {
    View {
        name: name.into(),
        title: title.into(),
        bounds: Bounds::new(0, 0, 80, 30),
        visible: true,
        state: Some(Box::new(SelectedState::new())),
        draw: Some(draw_toggle_button),
        layout: None,
        input: Some(input_toggle_button),
    }
}

fn draw_toggle_button<C, F>(
    e: &mut DrawEvent<C, F>,
) {
    e.ctx.fill_rect(&e.view.bounds, button_fill);
    e.ctx.stroke_rect(&e.view.bounds, &e.theme.fg);
 // rest of drawing code
}

fn input_toggle_button<C, F>(event: &mut GuiEvent<C, F>) -> Option<Action> {
    // get the view state
    if let Some(state) = event.scene.get_view_state::<SelectedState>(event.target) {
        // change it
        state.selected = !state.selected;
    }
    // mark it dirty
    event.scene.mark_dirty_view(event.target);
    Some(Action::Command("toggled"))
}

Next Steps

I built all of this so that I could make a tiny web browser for the LilyGo T-Deck. I have this code running in a different repo which makes use of this GUI repo. The repaint rates are reasonable but not great. So next up:

1. This lib needs a name. I could use some help on this.
2. I’m not happy with all of the dynamic casting that view functions need to do to access their own state. It feels like there should be a way to do this with just generics.
3. Everything is absolutely positioned. I’ve added an initial layout phase, but I haven’t worked out the details yet. Layout is especially important when we have themes and font size changes.
4. More built in widgets and GUI themes.
5. Examples beyond the T-Deck. I just bought a T5 E-Paper S3 Pro, so e-paper support is coming.

So.. I’ve still got more work to do. If you have questions or want to contribute please file a ticket or open a discussion question.

update

I removed the <C, F> generics everywhere. They were just for testing and I discovered MockDisplay, so they aren't needed anymore.

Keyboard

In some devices the keys and buttons are directly connected to pins on the MCU. This is not the case with the T-Deck. Since there are so many keys, and because it’s such a central part of the T-Deck experience, the keyboard is controlled by a secondary ESP32 chip connected to the main S3 chip by an I2C bus. To get keyboard events we just need to read from a specific address on the bus. The basic code looks like this:

let LILYGO_KB_I2C_ADDRESS: u8 = 0x55;

let mut i2c = I2c::new(
    peripherals.I2C0,
    Config::default()
        .with_frequency(Rate::from_khz(100))
        .with_timeout(BusTimeout::Disabled),
)
.unwrap()
.with_sda(peripherals.GPIO18)
.with_scl(peripherals.GPIO8);

info!("looping over the keyboard");
loop {
    let mut data = [0u8; 1];
    let kb_res = i2c.read(LILYGO_KB_I2C_ADDRESS, &mut data);
    match kb_res {
        Ok(_) => {
            if data[0] != 0x00 {
                info!("key {:?}", String::from_utf8_lossy(&data));
                delay.delay_millis(100);
            }
        }
        Err(e) => {
            info!("kb_res = {e}");
            delay.delay_millis(1000);
        }
    }
}

The firmware for the keyboard is minimal and buggy. It sends out ascii characters, but you can’t detect the shift or symbol keys. They’ve released an updated keyboard firmware but it seems cumbersome to install. I haven’t tried it out yet. Here’s another keyboard replacement firmware that looks interesting too. I’ve also not tried it.

Full reading keyboard example.

Trackball

The trackball is one of the T-Deck’s most unique features. It’s very odd, though. Real physical mice and trackballs use rotary encoders to give you changes along the x and y axis. On the T-Deck instead you get a pin to monitor for each of the four cardinal directions. It’s more like polling buttons. The standard code looks like this.

let tdeck_trackball_click = Input::new(
    peripherals.GPIO0,
    InputConfig::default().with_pull(Pull::Up),
);
let tdeck_trackball_right = Input::new(
    peripherals.GPIO2,
    InputConfig::default().with_pull(Pull::Up),
);
let tdeck_trackball_left = Input::new(
    peripherals.GPIO1,
    InputConfig::default().with_pull(Pull::Up),
);
let tdeck_trackball_up = Input::new(
    peripherals.GPIO3,
    InputConfig::default().with_pull(Pull::Up),
);
let tdeck_trackball_down = Input::new(
    peripherals.GPIO15,
    InputConfig::default().with_pull(Pull::Up),
);
let mut last_click_high = false;
let mut last_right_high = false;
let mut last_left_high = false;
let mut last_up_high = false;
let mut last_down_high = false;

info!("running");
loop {
    info!("button pressed is {} ", 
        tdeck_track_click.is_low());
    if tdeck_trackball_click.is_high() != last_click_high {
        info!("trackball click changed ");
        last_click_high = tdeck_trackball_click.is_high();
    }
    if tdeck_trackball_right.is_high() != last_right_high {
        info!("trackball right changed ");
        last_right_high = tdeck_trackball_right.is_high();
    }
    if tdeck_trackball_left.is_high() != last_left_high {
        info!("trackball left changed ");
        last_left_high = tdeck_trackball_left.is_high();
    }
    if tdeck_trackball_up.is_high() != last_up_high {
        info!("trackball up changed ");
        last_up_high = tdeck_trackball_up.is_high();
    }
    if tdeck_trackball_down.is_high() != last_down_high {
        info!("trackball down changed ");
        last_down_high = tdeck_trackball_down.is_high();
    }
    // wait one msec
    let delay_start = Instant::now();
    while delay_start.elapsed() < Duration::from_millis(1) {}
}

This code hooks up the four pins as inputs and then reads them every millisecond. This works but is a little clunky. The inputs are noisy and there is no suggested sampling rate. For use in a real application I would wrap this up into an async task with Embassy and use a channel to emit higher level mouse and scroll events. This is exactly what I do in my bigger projects.

Full trackball example.

Display

The display itself is easy to set up (if verbose). It works like any other SPI screen. Use the ST7789 driver from the mipidsi package like this:

use esp_hal::spi::master::{Config as SpiConfig, Spi};
...
use mipidsi::{models::ST7789, Builder};


...
// set TFT CS to high
let mut tft_cs = Output::new(peripherals.GPIO12, High, OutputConfig::default());
tft_cs.set_high();
let tft_miso = Input::new(
    peripherals.GPIO38,
    InputConfig::default().with_pull(Pull::Up),
);
let tft_sck = peripherals.GPIO40;
let tft_mosi = peripherals.GPIO41;
let tft_dc = Output::new(peripherals.GPIO11, Low, 
    OutputConfig::default());
let mut tft_enable = Output::new(peripherals.GPIO42, High, 
    OutputConfig::default());
tft_enable.set_high();

info!("creating spi device");
let spi = Spi::new(
    peripherals.SPI2,
    SpiConfig::default().with_frequency(Rate::from_mhz(40))
)
.unwrap()
.with_sck(tft_sck)
.with_miso(tft_miso)
.with_mosi(tft_mosi);
let mut buffer = [0u8; 512];

info!("setting up the display");
let spi_delay = Delay::new();
let spi_device = ExclusiveDevice::new(spi, tft_cs, spi_delay).unwrap();
let di = SpiInterface::new(spi_device, tft_dc, &mut buffer);
info!("building");
let mut display = Builder::new(ST7789, di)
    .display_size(240, 320)
    .invert_colors(ColorInversion::Inverted)
    .color_order(ColorOrder::Rgb)
    .orientation(Orientation::new().rotate(Rotation::Deg90))
    .init(&mut delay)
    .unwrap();

From here use the standard embedded graphics APIs to draw.

Full example code here.

Note that the display is mounted sideways so the height and width are swapped to 240 x 320.

Also note that I set the SPI rate to 40mhz which seems to be the max speed of the t-deck’s screen. I know that sounds fast but remember that SPI is a serial bus. Refreshing an entire 16bit 320x240 screen takes is 153.6 kilobytes per frame so base case, with no overhead, you can get about 30fps. In practice, it’s more like 20 and still has tearing. These devices don’t have any way to do accelerated drawing like lines or textures. (DMA might be a possibility) so be careful to draw as little as possible. I typically refresh my GUI every 50 msec (40fps) but only draw things that have changed. For example, here’s a simple block breaking game that only redraws the ball on most frames.

Here is the code to a little brick breaking game to show off fast graphics.

Touchscreen

The touchscreen is more interesting. The T-Deck has a real capacitive touch screen with up to (I think) five simultaneous touches. There’s an existing driver api for the GT911 chip that makes it very easy to use. Setup an i2c bus, init the driver, and loop to read touch points. Here’s a simple example:

use gt911::Gt911Blocking;

...
let mut i2c = I2c::new(
    peripherals.I2C0,
    Config::default()
        .with_frequency(Rate::from_khz(100))
        .with_timeout(BusTimeout::Disabled),
)
.unwrap()
.with_sda(peripherals.GPIO18)
.with_scl(peripherals.GPIO8);

let touch = Gt911Blocking::default();
touch.init(&mut i2c).unwrap();
loop {
    if let Ok(points) = touch.get_multi_touch(&mut i2c) {
        // stack allocated Vec containing 0-5 points
        info!("{:?}", points)
    }
}

See the full example code for more details.

That's all for this week. Next time we'll look at WiFi and HTTP requests. Have a great weekend!

Example Code

I'm a big fan of sharing research. After all, the only difference between alchemy and chemistry is showing your work. Sir Isaac Newton was both. Thusly presently I present to you: Rust Tdeck Experiments a set of examples for accessing the core features of the T-Deck. It includes

• drawing to the display with the embedded-graphics api.
• getting input from the keyboard, trackball, and touchscreen.
• scanning for wifi nodes and making an http request.
• listing files on an SD Card

I'm still working on the web browser, and of course a full UI toolkit required to implement it. It's not ready to show yet, but I can let you know I've been able to render CNN and NPR's lite weight mobile sites, even over SSL. Interestingly, TLS support is one of the hardest things to do on embedded systems. I suspect the security algorithms are slow and memory heavy.

Embedded Rust Evolution

Overall I'm pretty happy with Embedded Rust. It's fairly verbose but straight forward once you get the hardware configured. In typical Rust fashion, the code never just crashes. If you can get your code to compile then it will run. You can do fearless refactoring and not break anything (which I do frequently). All that said, the ecosystem is still pretty immature. The APIs have changed a lot over the past few years, rendering old code examples useless.

For ESP32 chips I'm using esp_hal, which is nearing its 1.0 release and stabilizing their APIs. I'm currently using 1.0.0-rc0 and will do my best to keep the examples up to date as the APIs evolve.

I hope this code will help you get started with your own embedded Rust project.

Cut through the hype

First I want to cut through all the hype. I’m dismayed that so much funding is going to ML and almost nothing else. What happened to all of the AR/VR startups? My beloved WebXR? But I digress. Back to ML.

A lot of the hype around LLMs stems from saying that we are just moments away from AGI. This is false. I get that the arguments are seductive. Breathless takes like ‘the fastest adoption of technology ever’ make us think that AGI is just around the corner. Moore’s Law lets technology scale exponentially and LLMs were only possible thanks to fast chips, therefore Moore’s Law gets us AGI. Right? No. No it doesn’t.

We have to remember that Moore’s Law doesn’t apply to anything other than transistors. Other tech will not scale the way chips did, unless you can turn it into a chip problem (which is why the mobile revolution seemed so fast and then petered out). Humans are very good at making the same thing over and over again. LLMs are not like transistors. LLMs require huge computational power to do their thing, but they also require massive amounts of data and clever algorithms to apply them. Those don’t scale exponentially. We are running out of public data. And even exponential growth can’t go on forever in the real world. There are always limits. Nothing is forever.

AGI beyond LLMs

LLMs are powerful but are already starting to show their limits. I don’t think a few more rounds of Moore's law is going to get us to AGI, despite what Sam Altman has said. Real AGI is going to take a different approach, combining LLMs with other kinds of systems. LLMs can’t correct themselves because they don’t have logical reasoning. They can generate something, but not know it’s correct beyond a statistical correlation with the corpus of the web it has ingested. These are called hallucinations (though perhaps bullshit is a better term). Humans make these problems too, but we can then check our thoughts: is the thing I just thought I saw likely to be real? LLMs don’t have the ability to say how confident they are. Humans can. Self correction is possible with code generation, because code can be compiled. You can run it to see if it actually does what it should. Anything where the answer can be easily verified is a good use of an LLM.

The Backlash is Coming

I see a backlash coming in the AI industry, and to avoid it we need to focus more on fail-safes and real value. Companies are making bold claims which will not only disappoint the bulls, but also anger the bears. Along the way we’re going to see the web filled with AI slop. Anything free will be flooded. Systems which depend on determining who is a real person or not are starting to break down. I don’t know of a solution beyond charging money. Maybe that’s a good thing. Massive unemployment due to LLMs probably won’t happen (long term we are in for a human labor shortage as the boomers retire) but if a recession is coming AI companies may get a lot of the blame. The backlash is coming.

A few weeks ago we saw a stockmarket correction that saw a sell off of AI and Nvidia stocks triggered by Deep Seek. This was inevitable. I never bought the idea that early AI companies could build a moat around their models. If one company can strap a bunch of GPUs together to train a model from the web then so can another company. LLMs are a technology that is destined to be commoditized. That doesn’t mean there isn’t money to be made; databases are commodities and Oracle is doing fine.

I think the real problem is that commoditization happened faster than people expected, and at the same time current AI apps just aren’t as useful and profitable as expected. This is okay. It has happened many, many times before in tech. It’s a bubble popping, or at least deflating.

I’m old enough to remember the dot-com bust. That didn’t mean the web was a bad idea. Just that these companies were too early and invested too much in technology that was about to be commoditized (fiber, servers, software). The two decades following the dot-com bust were the most productive our industry had ever seen. I think the same will happen with LLMs. It’s a new tool in the toolbox that we haven’t figured out how to use yet.

This doesn’t mean there isn’t value there. It’s simply that AI is a technology, not a product. It’s going to take longer to find the value than expected. The current LLMs just aren’t reliable enough. Over half of LLM-written news summaries have “significant issues” according to a recent BBC analysis.

Things I’m Not Worried About

I’m genuinely not worried about excessive power usage from AI. Deep seek is showing that LLMs can be trained with less computation and energy. Overall power usage for the economy will continue to go up over the next decade, but at a rate consistent with historical averages. AI is a blip and the power company stocks will fall back to earth soon. Power usage is an engineering problem. It’s the kind of problem we know how to solve. Very little work has been put into making these AI systems power efficient yet. I expect that to change over the next few years.

I’m not actually too worried about LLMs being controlled by just a small number of companies. The technology is proving easy to duplicate at scale. Even if we don’t have open models from the big US companies, we likely will have them floating on the Internet sourced by Chinese companies, and efficiency gains will let us run them locally.

I’m also not worried about AI alignment and robots taking over. It’s important to remember that these things aren’t alive or conscious in any meaningful way. I’m not worried about ‘alignment’ because the question itself assumes intent and intelligence that these things don’t actually have. I am worried about ‘bias’, though, the same as I’m worried about bias in psychological studies whose core sample population is Ivy League psych majors. We already have alignment problems with other non-human processes. They are called corporations. What does it mean for a business to be ‘aligned’ with human interests? AIs will need be subject to regulation the same way businesses are. Imperfect, to be sure, but not an existential crisis.

What we should be working on

LLMs are a technology, just like transistors and lasers. They are not products. We haven’t seen the real AI products yet. I feel like the current problems are largely UI and product definition issues. LLMs work best where the cost of failure is low and where a human can review the results.

The problems come when we trust them to work without supervision. We aren’t designing these things to fail, meaning to properly handle failure. Failure will always come. If you don’t handle it you are just doing bad engineering. Consider a company replacing a call center with a chat bot which can’t actually solve the 5% of problems people call human support centers for. It’s not allowed to. Or insurance review boards replacing reviewers with LLMs that fail in some cases, and no way to go back to a human to address the failures. LLMs work best where the cost of failure is low. Is the generated essay too wordy? Just generate it again. Does this scan indicate cancer? Maybe, but we need to delegate to a human for the next step.

Right now we put too much faith in these systems. The computer should never make a promise it can’t keep. Above all, the engineers need humility. Solve the small part of the problem that you can, and be humble that you can’t solve it all and account for it. It’s going to require a lot of work. A lot of experimentation to find the cases where AI provides real value instead of cheap-to-generate slop. Things may get a lot worse before they get better.

Conclusion

We haven’t seen the real AI products yet. Most tools are still just generative. They can’t advise me on what I already have. They don’t have judgment. Where are the tools which analyze my code dependencies and figure out what to swap out with slimmer replacements. But we will get there. Where is the LLM powered security analyzer that can find bugs in my code beyond the heuristics. What would it look like to have a word processor for thinking? Something helps me organize my thoughts, not just rewrite them. An AI that was a trusted research partner. AI as a collaborator, not a replacement. We are a ways away from that still. We need to augment humans, not replace them.

If you like my ideas and are hiring at your AI company, let me know.

WebXR

Mixed Reality (MR) refers to the spectrum of interfaces between and including Virtual Reality (VR, fully opaque) and Augmented Reality (AR: fully transparent). The Mixed Reality Group was a team at Mozilla researching MR, building web standards around it, and making a MR first browser for multiple MR devices. At one point we had tech running on the Oculus Go, Oculus Quest (1), MS’s HoloLens, and some Android-derived devices like the Pico (who I just learned was acquired by Bytedance).

At the center of all of this was WebXR: a set of web standards (actual W3C standards) for building and securing mixed reality applications on the web. And I got to be a part of it. My team and I built developer tools, epic demos, wrote tutorials (since scrubbed from the web), and promoted WebXR content. It was a thrilling time and a highlight of my career.

The failure of WebXR

After the WebXR group was shut down at Mozilla a lot of other MR projects have been canceled, scaled back, or put on the back burner. In the following five years MR for the open web hasn’t happened and WebXR is largely DOA.

At Mozilla, after laying off most of the Emerging Technologies division in 2020, Hubs survived but mostly for the non-immersive cases, and even that was cut last summer. Everything else is gone. Our blogs are gone. The github repos have been archived. Almost wiped clean from the face of the web.

Apple supports the WebXR standard but only on their VR headset (more on that in a bit). We (Mozilla) had it AR based WebXR running smoothly on iPhones, but Apple never unlocked the API in Safari. In fact, Apple’s native ARKit efforts seem to have completely dropped off the face of the earth. They are still supporting it and occasionally add new features, but don’t highlight apps built with it or use it any way I can see.

Microsoft was a WebXR contributor but they killed off their Windows MR initiative and HoloLens 2 is stuck on an OS that will soon exit support and they have shown no indication there will ever be version 3.

Google abandoned Daydream and has done nothing with Android’s AR APIs. They also shut down multiple web-based systems that supported AR (visual tours of maps, a 3d model store, and I think a few others). Chrome itself still supports WebXR, presumably on any PC based VR headsets, but they don’t seem to do much work on it anymore.

Meta still supports WebXR on the Quest, but I never hear about websites being built to use it anymore. They don’t advertise it or seem to encourage development.

So MR on the web hasn’t happened. Or it happened and then it died. This makes me sad. Just writing this section of the article makes me sad. I really wanted to build the metaverse as an open network of websites that visitors could jump between. To make immersive a native part of the web. In the end.. maybe the world just didn’t want it

Beyond the Web

The non-web MR platforms have slowly improved but not matching what we dreamed about. Meta shipped the Quest 3 (and the expensive Quest Pro, for some reason) and it has a few popular games and exercise apps, but it hasn’t grown into a thriving platform beyond games. They also tried to build a VR social network but if none of my VR using teenagers have even heard of it then clearly it flopped.

Sony has lost interest in the Playstation VR2, but that was always a closed platform anyway. Pico has continued to make headsets but I don’t think they are sold in the US. And of course Microsoft and Google shut down their own offerings.

Google recently announced Android XR, but I question if it will go anywhere. Google has a habit of starting big initiatives and then abandoning them when they get bored. At least that’s what it looks like from the outside. I imagine internal politics are different. I’d love more information here.

Apple Vision Pro

And finally, a latecomer to the game (or maybe still too early) is the Apple Vision Pro. It’s amazing hardware but doesn’t do much and is very expensive. How is it that **the best designers at the wealthiest company in the world can’t come up with better uses for AR than 3D photo galleries and VNC?**And if that’s the best they’ve got, then why did they release it at all?

Usually when Apple introduces a new technology they have some amazing showcase uses to drive adoption and excite developers. However, sometimes it’s simply impossible to nail the killer uses without a bunch of real people trying it out in the real world and seeing what works. The Apple Watch was like this.

The first versions of the Apple Watch were slow, had terrible battery life, and couldn’t do very much. In most ways they were worse than everything else on the market. The one true bit of value they added was notifications from your phone on your watch, which ended up being easy for others to replicate anyway thanks to BLE. But Apple didn't give up.

Apple’s Watch OS team pivoted the UI and core uses several times before nailing the wellness functions around version 5. So why did they ship before then? Why did watches 1-4 exist outside the lab? Because without them they could never get to Watch 5.

I think, or at least I hope, Apple Vision Pro is like this. There will have to be a few crappy versions until we get to the good one that has the right mix of weight, features, and battery life; and has a few (or even just one) truly killer use that justifies the product.

I’ll say this about Apple. They don’t give up. They keep iterating until they nail it. Of course, once they find product market fit they sometimes let products atrophy, but that’s a story for another day.

A different path forward

Maybe evolving VR to AR was the wrong approach. Maybe instead we should focus on regular glasses with a heads up display and actual useful functions before trying to nail the real world overlay and occlusion problems. Meta’s RayBan smart glasses are far more interesting to me than either of their separate VR or AI initiatives.

Maybe we need an AI agent looking over your shoulder analyzing what you are doing. Being the guy in the chair to our superhero selves. Secure AI agents could be huge here.

To anyone hiring in the VR and AR space, please get in touch. I’d love to work in Mixed Reality again.

Going Back in Time

I’ve worked in a bunch of different areas of tech during my 30 years of being a professional software engineer (Holy carp! I’m old!). From UI toolkits to network attached storage, from mobile applications to low latency networking. Though the jobs have been diverse, there are some common themes, especially human computer interaction. This should not be a surprise as Graphics, Visualization and Usability was my specialization in college. Still, it’s interesting that I keep coming back to the area over and over.  It’s surprising that my initial instincts at 18 were actually correct.

Let’s go through some current tech areas that are heavy with HCI and are growing.

Artificial Intelligence

I am not an AI maximalist. I don’t believe AIs are conscious, nor do I think we are anywhere close to AGI. Sam Altman seems to think we are, but he carefully hedged his bets by saying we will have super intelligence within a few thousand days, which is a clever way of saying “at least a decade away”. I personally think it will be several times that, assuming it is possible at all. In either case, it doesn’t affect my job hunt today.

However, intelligence and consciousness are different things. It is entirely possible to have intelligent machines that are not conscious, and that is probably preferred anyway.  I’m not even talking about future tech. We have yet to absorb the possibilities of current LLMs, much less the ones coming down the pipe. Lasers were first built in 1960 and we are still discovering new uses for them.

I feel that the next steps for AI are largely a user interface problem. These things are powerful but flawed. Making them applicable to more situations, and making them actually help humans instead of replacing them, is going to require some deep UI work. This is something I want to work on.  If you are in the AI industry and looking to hire someone on the UX side, give me a call.

I’m going to explore this deeper in its own post.

XR: Augmented and Virtual Reality

Working in the Mixed Reality group at Mozilla for three years was probably the best work experience I’ve ever had. The end of that group was devastating for me. I got to build cool demos, developer tools, write blogs, as well as be on the W3C standard committee to make XR widely available and defend against attacks that could fill many a Black Mirror episode. It was a thrilling position and I really miss it.

Since 2020, when Mozilla dissolved that division (along with essentially every other group outside of the core browser), XR has largely been a disappointing industry. Meta has continued to improve the Quest but it hasn’t broken out of gaming. Microsoft shut down Windows MR. Google abandoned Daydream. And finally Apple’s Vision Pro has been a huge disappointment. While the hardware is incredible, how is it that the best designers in the world can’t come up with better uses for AR than 3D photo galleries and VNC?

Maybe evolving VR into to AR was the wrong approach. Maybe instead we should focus on smart glasses with a heads up display and actual useful functions before trying to nail the real world overlay and occlusion problems. 

I’m cautiously excited to see Google return to the immersive space with Android XR, and Meta’s RayBan smart glasses are far more interesting to me than either of their separate VR or AI initiatives.

More soon in a standalone post.

Embedded Hardware

The capabilities of modern embedded systems amaze me. Anyone can make fairly complex hardware using free circuit layout software, embedded programming languages like CircuitPython, easy to acquire components (I'm looking at you RP2040), and cheap PCB manufacturing services. Common standards like USB-HID make it even better. And yet..

Theoretically the low barrier of entry should trigger a Cambrian explosion of bespoke hardware, but I haven’t seen much of that yet. All keyboards look the same. All phones look the same. Where's the crazy hardware?! Yes “hardware is hard” but it’s a lot easier than it used to be.  Where's my LLM powered micro-robots? How are vacuums and airpods the only new consumer hardware category of the past decade?

I suspect there is a lot interesting things happen that just aren't visible yet. One of my personal product ideas is a LEGO compatible macropad, which I’ve prototyped and am considering bringing to market.  I’m also looking at building a programmer alarm clock, pixel style wall displays, and other products that will improve my industrial design skills. None of these ideas are likely big enough to be my full time job, but they’d make for some interesting side projects.  More on this later.

Desktop Operating Systems

I spent 5 years on the desktop Java UI team at Sun, and at least another five years before that building lots of desktop software. I remember the Gnome vs KDE wars. I eagerly awaited every new release of OSX. Those days are over.  

From a UI point of view desktop/laptop OS dev is dead. All of the effort is targeted at mobile while the desktop OSes have degraded from increasing monetization, API neglect, and centralized control; all at the cost of usability and productivity.  A new OS from scratch could be an order of magnitude more productive with far fewer resources if it was designed around simpler messaging APIs and a database filesystem.

Sadly, while I feel there is desire for a desktop OS that is as usable and productive as, say, early OSX and Windows XP, with modern architectures and hardware; I don’t think there is a way to sell it.  I don’t know how to build a business or open source project that could fund professional development of a new operating system (even if it used the guts of Linux to get started).  I have, however, continued some personal research on this topic which I’ll cover in my deep dive on OS and UX dev. I just wish I could find a way to turn it into an actual job.

No Code / Low Code

AKA: coding for the masses. Once upon a time there was a lot of research and product development in software to let non-programmers be productive with computers. To let normal people create their own solutions.  Hypercard. Visual Basic. For a variety of reasons little progress has been made for several decades. It’s sad to me that spreadsheets are still the best end user programming system we’ve come up with. That Apple's addition of math evaluation to Notes is such big news represents how lackluster things have become.

LLMs may be the game changer here. They provide a fundamentally new way of interacting with computers that I think could let people really solve their own problems. I don’t know how to build a business out of it, but there’s clearly something interesting going on here.

To be clear, I don’t think that all programmers will be unemployed because LLMs can generate code for you.  Code generation is not the answer.  Building a secure online application to collect data is far more complicated than it should be. However, asking an LLM to write the same code that a human programmer would is a sure fire way to get an insecure unmaintainable mess. I suspect the right answer is some new model of computation more amenable to the fuzziness of small bespoke applications, and perhaps a new kind of programming language to go with it.

Again, I don't know how to make a business around this, but there's some interesting kernel to be explored here. Contact me if you'd like to talk about it.

Medical Software

The modern medical software that patients interact with (health record systems, patient portals, etc) is a mess. It is some of the worst software I’ve ever had to use (and sadly I’ve had to use it a lot lately). Unfortunately, I strongly suspect that the root cause is that the medical system itself (at least in America) is a mess. Software reflects the structure of the organization that built it.  I’d love to help fix medical software, but faster billing systems won’t fix broken business models. I don’t know what the solution is here but as someone currently wading through a new diagnosis I’d love to help fix it.  Please contact me if you’ve got some medical startup opportunities.

Technical Documentation

Writing technical documentation is still harder than it should be and too locked into proprietary software. Writing good docs was the core of my last job, so I’m familiar with a lot of the problems.  While I love the idea of GitBook, the reality of WYSIWIG editing Markdown for large documentation sets is still buggy and frustrating. I have some product ideas and GUI prototypes here that I want to explore in future posts, so I’ll save it for then.

Next Steps

So where am I going with all of this?

Right now my plan is to explore some of these ideas through a small side company (I have a certificate of existence!), while also continuing to get healthier and look for a new full time position.

Look for my deep dive articles coming soon and stay healthy.

Tiny Apps

One of the keys to IdealOS actually being “ideal” is having apps which are tiny. They should have very little code. The less code there is the less space there is for bugs to hide. Tiny apps also tend to run faster and be easier to maintain. If we assume an always accessible database, then theoretically a lot of the complexity of writing an app can go away, at least if we have the right abstractions.

To that end I’ve been prototyping a React library that lets you define a data schema, then use it as the core data structure in your React app without having to manage state updates, and automatically persisting to local storage, resulting in an extremely compact app.

Runtime Types

The core idea I'm prototyping is: make a schema using using prototype objects then clone those for your actual instances. Typescript is a compile time type system, but my prototype lets us do all of the type calculations at runtime. Schema objects contain all of the information for other code to read the entire schema, generate UIs based on those schemas, load and save to JSON, and do other useful things.

The schema is composed of a few core object types: atoms, lists, maps, and... Actually, it would be easier if I just show you.

Here’s a simple example of a todo list item:

const TodoItem = makeMap({
    title: makeString("untitled"),
    completed: makeBoolean(false),
})
type TodoItemType = typeof TodoItem
const TodoList = makeList<TodoItemType>(TodoItem)
type TodolistType = typeof TodoList

Now lets create a list with two items in it.

const data:TodolistType = TodoList.clone()
data.push(TodoItem.cloneWith({
    completed: false,
    title: "make breakfast"
}))
data.push(TodoItem.cloneWith({
    completed: true,
    title: "buy milk"
}))

From this list we can pull out the data, manipulate it, loop over it, and all of the other things we normally do with data structures. All items have a built in toString() method so we can easily print it as well.

console.log(data.toString())

List:type_40070(make breakfast,false), 
     type_40070(buy milk,true)

Now let’s create a simple React component to view the list of items, and add, edit, or delete them.

export function TodoListExample() {
    const [selected, onSelect] = useState(()=>data.get(0))
    const addItem = () => data.push(TodoItem.cloneWith({
                                         completed: false,
                                         title: ""}))
    const deleteItem = () => data.deleteAt(data.indexOf(selected))
    return <VBox>
        <HBox>
            <Button title={"Add"} onClick={addItem}/>
            <Button title={"Delete"} onClick={deleteItem}/>
        </HBox>
        <ListView data={data}
                  selected={selected}
                  onSelect={onSelect}
        />
    </VBox>
}

That’s it. That is all of the code that we need to make a simple todo list. Rendered it looks like this:

Events and change propagation all happen internal to the structure, so your app doesn’t need to manage updates when new items are added. It will just do the right thing.

ListView

The ListView component knows about these smart objects, so it can render and select elements from the list right away. When something changes in the list, or a property deep down inside the list’s elements changes the ListView will be notified and redraw itself automatically.

ListView does not know, however, what our TodoItem object actually is so cannot render the items the way we want. By default it will call item.toString() which recursively calls toString() on the child properties. That is the output we saw above.

The next step is to tell the ListView how to render TodoItems using a renderer. If we just want a different string representation we can give it a StringRenderer like this:

const SimpleTodoItemRenderer:StringRenderer<TodoItemType>
 = (item:TodoItemType) => {
    return item.get('completed') + " " + item.get('title')
}

This works if we just want text output, but more likely we want to be able to edit the todo item from the view. In that case we create a ListItemView renderer with a checkbox for the completed property and an EditableLabel for the title.

const ComplexRenderer: ListItemRenderer<ItemType> 
   = (item: ItemType) => {
    return <>
        <BooleanCheckbox value={item.get('completed')}/>
        <EditableLabel value={item.get('title')}
                    strikethrough={item.get('completed')}/>
    </>
}

Again, these components are aware the smart objects. BooleanCheckbox knows how to toggle the boolean item.completed property. The EditableLabel knows how to edit a text property. All of the event propagation is handled internally.

With the Right Datastructure the Code Just Falls Out

So where are we now? We have the ability to define a datastructure out of lists, maps, and typed atoms like strings and numbers. Then we have smart react components know how to intelligently render the data structure and generally do the right thing. This is all pretty cool but we could have done this with regular TypeScript objects. The magic is that we have the types available at runtime. We have a schema. Because of the schema other features fall out for free:

Serialize

Save to JSON with data.toJSON() which recursively saves the data structure to a JSON object.

Deserialize

Restore from a JSON object with TodoList.fromJSON(json).

Property Defaults

Provide default values when adding new fields to a type.

const TodoItem = makeMap({
    title:makeString('untitled'),
    completed:makeBoolean(false),
    cost:makeNumber(10)
})

When restoring from JSON, if the stored object doesn’t have a cost property, the system will automatically use the default value of that property. Calling cloneWith on Map objects will also use the defaults if the property is omitted.

// completed and cost are set to defaults
const todo2 = TodoItem.cloneWith({title:"hello”}) 

History / Undo & Redo

We can monitor all changes to a datastructure by attaching a change handler to the top node. This means we can record every change in memory and undo those changes with fully generic code. I created an AHistory object which does exactly that.

const history = new AHistory<TodoListType>(data)
// add item
data.push(TodoItem.clone())
// undo the add
history.undo()

Constraints

The runtime schema also gives us constraints that can’t be expressed directly in the type system. Suppose we are making a drawing program and want to express that the radius of a circle must be a positive numeric value. We can do that with a constraint function that must return true for the change to be allowed.

const GreaterThanZero:Constraint<number> = (v) => v>0

const Circle = makeMap({
    x:makeNumber(0),
    y:makeNumber(0),
    radius:makeNumber(1,{
        cons: [GreaterThanZero]
    }),
})

Now the Circle type will never allow the radius to be set to something less than or equal to zero. UI components can use this information as well. A smart number editor could use a numeric input field with min set to 1 to disallow entering an invalid value.

The Missing Piece

One thing I haven’t talked about is: where does the data come from? For something document oriented like a drawing program, we can imagine the document is represented as a tree structure which is persisted to a JSON file or exported as an image.

One of the core features of IdealOS was that it was database oriented. Sure, we could store a document as a single object in the database, but part of the power of a db is the ability to search and link. Right now the TodoList object wraps an array in program memory, but it doesn’t have to. What if, instead, TodoList wrapped a live query in the database. Then the program wouldn’t have to consider persistence at all. Data comes transparently from the system database into the UI. Editing a field would automatically update the database. Most applications would not need code for saving to disk or interacting with the database. The data structures would keep themselves up to date, letting the app focus on doing one thing well: helping the human interact with data.

Conclusion

The core idea of a schema, or runtime type information, is very powerful. We can build reusable components that automate away much of the busy work involved in making a GUI. The GUI can customize itself to the structure of the data rather than vice versa. Combined with a built in database, the applications for IdealOS can be tiny and incredibly easy to build. We can even imagine letting users build their own apps visually for IdealOS, resulting in an incredibly flexible and hackable system.

You can find the source to this unnamed db object library https://github.com/joshmarinacci/tool-toolkit.

New Screens

The WaveShare board I'm using has a lipo battery charger inside it, but until now I hadn't exposed it. There is now as screen that shows the current battery percentage, though I think my calculations might be off. What do you think?

I also added a screen with a timer. The actual timer itself works, but it won't alert you when the timer ends if you switch to another screen. I need to figure out some sort of a background task.

I also added a moon phase indicator using some calculations I found on the web. Again I'm not sure it's accurate, but the fun part is the star field particle effects behind the text. The hard part was figure out how to draw lots separate pixels (one for each star) without forcing the entire screen to refresh. The secret is to call bitmap.refresh() after each star is moved, rather than after drawing all the stars. If you do all the stars then you get one giant dirty rectangle that fills the whole screen and will be slow. If you call refresh after only changing two pixels (undrawing the old, then drawing the new), then in most cases you'll have a dirty rect that is only 2 or 4 pixels in size, which is super fast to draw. Even with 50 particles, the total pixels drawn with my incremental approach is still a fraction of the full 240x240 screen size.

Sleeping

The final big feature is making the watch sleep, though it's only partially working. The screen will turn off after N seconds of no input. This more than doubles the watch lifetime on a single charge. However, the touch sensor and CPU are still running a full power, only the screen is off. The device does have a more advanced low power mode that will wake up the main CPU when a tap is detected, but unfortunately it requires monitoring a specific interrupt pin, which CircuitPython doesn't support. There is some async io support in the form of a counter api, that I plan to investigate next.

Pausing

I'm happy with the progress of my watch, but I'm pausing this project for now while I work on a few other electronic projects. I recently built a set of animatronic eyes based on Will Cogley's video tutorial, which I plan to expand into a singing tiki head. I'm also experimenting with the new synthio CircuitPython API to build a step seqencer in a gameboy-like form factor.

As always, the full source is on github.

Icon Fonts

Now let’s suppose you want some nice icons. You could download each icon as PNG or SVG and then convert to CircuitPython’s preferred BMP format, but icon fonts are a thing. Let’s use them. Google created the Material icon font for use across all of Android and the Web. They are completely open source and have icons for pretty much everything. We can’t use it directly, however, because it’s in what’s known as a variable webfont format which can have multiple weights and styles in a single file. Instead we want the static truetype fonts from the GitHub repo. I chose the font called MaterialIcons-Regular.ttf.

When you download a single Material icon font you’ll see that it is huge. 348KB for a single weight! That’s because it contains over 3000 icons. 348KB is too big for our little RP2040 that has less than 200KB, and of course we don’t need all 3000 icons. In most cases we only want five or ten of them. So we need a way to subset the font.

We will use a python program called pyftsubset that knows how to subset fonts. It is part of the font-tools. Install font tools with pip then run pyftsubset to make sure it’s installed correctly.

pip install fonttools
pyftsubset --help

Now let’s create a new icon font with a subset of the Material Symbol Font containing only the glyphs we want. If you select an icon back on the Material Symbol Font webpage a sidebar will appear on the right. At the very bottom it will list the code point of that particular icon. That’s what you need. Write down this code point, and the points of any other icons you want to extract. For my example I want the icons for Settings (e8b8), Battery Full (e1a4), and Timer (e425).

pyftsubset MaterialIcons-Regular.ttf --unicodes="U+e8b8,e1a4,e425" --output-file=icons.ttf

You can add the --no-ignore-missing-unicodes option to print an error if you typed in a code that doesn’t exist in the font. The resulting font is still in TrueType format, so we need to convert to bitmap to use it.

otf2bdf icons.ttf -o icons.bdf -p 16

And now we can use it in our program. To use the icons we have to specify the codes for each icon with unicode escapes:

time_label = Label(
    font=icons,
    text="\ue8b8\ue1a4\ue425",
)

Here it is on my prototype device.

C’est magnifique

Animation in CircuitPython

For my first attempt I captured the JS animation as a video, scaled it down to 240 x 240, then converted it to an animated gif. On the device the gif renders at 2 or 3 fps. A far cry from the smooth 60fps of my browser.

For my second attempt I exported a background image of the disk, drew the eye on top, then animated the eye lids as rectangles that move from the top and bottom to the center. It was still rough, but seemed to be maybe 6 or 7 fps.

What's going on here? While this device is slow, it's stil a 133mhz process. I had far faster animation back on my old pentium, even when running Java code instead of optimized C. The problem must be in how the display is updated. After spending the next 8 hours researching how these sorts of displays work, and how CircuitPython exposes them to the programmer, I've come to this conclusion: Tiny TFT displays are just slow by nature, even with raw C code, but there are ways we can speed it up. That's what the rest of this post is about.

How CircuitPython Does Graphics

CircuitPython can be quite fast because all of the performance sensitive work is implemented in optimized C code. This includes drawing. The displayio system lets you use bitmaps with indexed colors to save memory, which means copying bitmaps back and forth can be fast. Drawing shapes can be almost as fast as C because we are just setting pixels in an in memory bitmap, and of course things like vertical lines and rectangles can be optimized using memcpy routines. Actually drawing to the screen is a different story. It can be very slow.

No matter how you draw into the framebuffer, that buffer has to be uploaded to the display a bitmap. Most of the little hobiest displays use a serial bus called SPI. Looking at the source to the driver for my board, the only thing custom was the init code. All uploading is a standard FourWire connection, which is a type of SPI connection.

I’m using a 240x240 16bit color screen connected over SPI. It has no indexed modes or accelerated drawing routines (as far as I can tell none of these hobbiest screens do). This means to do a full screen refresh requires sending two bytes for every pixel on the screen, over a serial port, one bit at a time. The DisplayIO routines are smart and try to use batches, but even if you set the entire screen to a single color, that is still a lot of data to transfer.

240 x 240 x 2 bytes per pixel equals 115200 bytes per screen, or 912000 bits per frame. For 30fps that’s over 27 million bits per second with zero overhead. In practice I’m guessing we get half of that. A lot of SPI ports only run at 10 or 20 MHz, so getting a consistent 30fps just isn’t going to happen. That’s just the nature of SPI. Desktop computer screens use protocols that can send more than one bit at a time, and operate much faster than 10Mhz. SPI just can't.

There is one saving grace, however. These screens have an internal memory buffer and support partial refreshes. This means you can update rectangular subsections of the screen containing just the changes between the last frame and the next. It won’t help for full screen animation, but for typical GUI work where only a few things change we can make it quite performant. We just have to go back to rendering algorithms from the 90s (like the Java Swing toolkit that I worked on for 10 years). This is in fact what displayio does.

The inability to do smooth full screen refreshes does mean certain types of effects, like slideshow animations and or fading the entire view to black are impossible. (Well, maybe we could animate the backlight?)

Earlier I said there are not acceleration commands, like RLE encoding or bitblts, or indexed color palette swaps. This is true. However, this particular chip does have vertical scroll offsets, which might make it possible to do smooth vertical scrolling of at least part of the screen. I’ll have to look into that later.

Improving Performance

So to start with, the only perf improvement we can do is set the SPI bus to the fastest possible speed. In the sample micropython code that comes with this screen I *think* they are setting the speed to 100mhz but the C code seems to be 40mhz and the Arduino code 66mhz. 🤷 Switching from 20mhz to 100mhz seems to speed up the animation I’m working on, but really I need a proper benchmark to measure it.

I created a little script to redraw a fullscreen bitmap over and over, with different SPI speeds. At 1mhz I’m getting an 690ms per frame or about 1.45fps. As I increase the speed the fps improves linearly until it tops out at 64 MHz for a frame time of 144ms or just under 7fps. Setting the speed any higher has no effect. Calculating the multiplying the size of a frame in bits times the fps gives me 6.4Mbps, which sounds correct for an uncompressed video stream. This seems to be the max we can get.

Incidentally if I leave off the speed setting it seems to default to 20Mhz. If I leave off the target framerate of 60 it drops to 3.66fps. If I set the target framerate to 10 (which is still higher than our max) it drops to 5.38fps. Setting the target to 60 lets get back to our max fps of 7. So I’m not sure what exactly the target framerate is doing but it seems to have some inaccuracies. Perhaps it’s allowing for more GCs? Setting it to 100 actually seems to increase the fps slightly to 7.09. Setting it to 1000 increases it some more, but only to 7.3. So clearly there is some tradeoff here but if it’s impacting battery or heat then it’s not worth it for that tiny speed improvement.

Partial Refresh

Now let’s try filling just part of the screen and see if we can get speedups?

Sure enough: halving the pixels roughly doubles the frame-rate. Cutting them to a quarter gives us a 4x speedup. So 25fps seems pretty good, right? Yes and no. The numbers say it is pushing that many frames per second, but visually it does not seem to actually be rendering that fast. In fact there is a lot of tearing. Recording with my slow-mo camera shows that it takes about 70-100ms to render a frame, or in the range of 10-14fps, even though we are sending significantly more frame data than that. Technically we could refresh at 64x64 rectangle over 60fps, assuming we had no overhead, but that’s only 16th the number of pixels. And visually it doesn’t look that fast, plus processing that much data in code for something like a video would be insane.

Still, this does give us something to work with. A full screen refresh is around 7fps, but if we only want to refresh part of the screen at a time then we can get 15 to 30fps fairly easily.

Let’s try doing several circles rotating. I can get a calculated fps of over 30 fairly consistently. In fact, if I set the target framerate to 20 and the speed to 200Mhz I get an extremely consistent exactly 32fps. Something must be syncing up nicely here. I'm not doing anything clever in my code. I created some circles and move them every frame. DisplayIO must be doing some dirty tracking underneath and only sending the smallest number of changed pixels to the screen. That's how we get 30fps. But again, visually it does not seem as smooth and there is some tearing. I suspect the refresh rate of the physical LCD is lower than how fast we can pump out screens. It also looks a little choppy because my circles are restricted to integer coordinates. I can't move a circle to be on the edge of two pixels and get anti-aliased drawing like I would on HTML canvas. But for what I'm building I can live with these constraints.

Next Time

So what have we learned?

1. All of my research taught me that the SPI bus is the bottleneck, and there's no way to get more than 7fps for a full screen refresh. However, partial screen refreshes can quite fast and the displayio API was designed to enable this. They've done a great job. I don't think I could do better without hardware changes.

2. Make sure you set the SPI bus to it's max speed. The default may be much lower than the component is capable of.

3. Design your graphics an animation around updating the fewest number of pixels

4. Use the backbuffer to your advantage. You can make complex graphics as long as the number of pixels changed per frame is small. Cool particle effects should be possible with this technique.

Next time I'll show you some graphics examples I've been working on as well as how to access the touch events to make a little painting app. All of the source code for this project, including my performance testing on in this github repo.

First boot

After plugging it in the device shows a boot screen, photo of some greenery, and then this live sensor output screen; highlighting the built in accelerometer, gyroscope, and battery sensor.

In the photo you can see just how small this thing is. 1.28 inches across! That's a USB-C connector at the bottom there. The screen has a 240x240 pixel display that I found to be pretty sharp. It looks a little blurry here because I hven't taken off the plastic protective film yet.

On the back there are two buttons for boot and reset. When I plugged it into my computer it did not come up as a local drive. When I held down boot while plugging it in my Mac asked to trust the device and a tiny drive called RPI-RP2 popped up. The bootloader text shows:

UF2 Bootloader v3.0
Model: Raspberry Pi
    RP2
Board-ID:
    RPI-RP2

So far so good. After dragging on the CircuitPython UF2 file to the device it rebooted and came up as a CIRCUITPY drive. Using screen /dev/tty.usbmodem2101 got me to the python shell. Success!

Drawing to the Screen

I struggled to draw to the screen for a few hours. Some research indicated this device uses the gc9a01 graphics driver, which is supported by the latest CircuitPython release. None of the sample code I found worked, however. The backlight would come on but nothing was drawn. By coomparing to older MicroPython code by Alasdair Allan, I confirmed I was using the correct pins. Even the main spec page says I am.

After some more searching I came across this other data page specifically for the touch version. Nothing seemed awry, but looking at the schematic diagram I could see the pins are slightly different than the non-touch version, and different from the docs in the demo code.

In the end having the touch version meant one pin was different. Below is the code I ended up using. Notice that reset is not set to LCD_RST as would be on the non-touch device. Instead we have to use pin 13.

spi = busio.SPI(clock=board.LCD_CLK, MOSI=board.LCD_DIN)
# LCD_RST is 12 in regular version
# but we need 13 for the touch version
display_bus = displayio.FourWire(spi, 
   command=board.LCD_DC, 
   chip_select=board.LCD_CS,
   reset=board.GP13)
display = gc9a01.GC9A01(display_bus, 
   width=240, 
   height=240, 
   backlight_pin=board.LCD_BL)

And with that everything worked. Here's a photo of the LCD showing a indexed bitmap of Earth.

Next Time

Now that I have the board working it's time for some fun. Next up I want to get print a case for it, access the accelerometer, and build some sort of interaction app. Maybe a puzzle game.

Adapting for High DPI Screens

Now let’s consider HiDPI. Suppose you want a 500 x 500 pixels canvas, but if the viewer has a display with 2x HiDPI support then it should scale it up to look the same, but sharper. We can do this by increasing the pixels in the canvas and using CSS to constrain it back to its 500 x 500 pixel size. Remember that in CSS pixels are virtual pixels. They are scaled by the browser automatically. So on a 2x screen we’d want the canvas to be 1000x1000 image pixels but scaled back to 500x500 virtual pixels, which will be scaled back up by the browser to 1000x1000 physical screen pixels.

<canvas id="can" width="1000" height="1000"></canvas>
<style type='text/css'>
    canvas {
        width: 500px;
        height: auto;
    }
</style>
<script type='javascript'>
    const ctx = document.getElementById('can')
    ctx.fillStyle = 'red'
    ctx.beginPath()
    ctx.arc(250,250,250,0,Math.PI*2)
    ctx.fill()
</script>

Left is without the fix, right is with the fix.

Notice in this zoom how the pixels are sharper in lower image

Of course the ratio might not be 2x. There are some mobile devices with 3x screens. Also, the canvas now thinks it has 1000px of space, but if your drawing code assumes 500px it will will only draw in the upper left corner of the canvas. Let’s scale the size of the canvas to use the real device’s DPI in code, and then reverse the scale for drawing the circle. The property window.devicePixelRatio will give us the correct scaling factor.

<canvas id="can" width="500" height="500"></canvas>
<script type='javascript'>
    const canvas = document.getElementById('can')

    let dpi = window.devicePixelRatio
    const WIDTH = 500
    const HEIGHT = 500
    canvas.width = WIDTH*dpi
    canvas.height = HEIGHT*dpi
    canvas.style.width = `${WIDTH}px`
    canvas.style.width = `${HEIGHT}px`
    const ctx = canvas.getContext('2d')
    ctx.save()
    ctx.scale(dpi,dpi)
    ctx.fillStyle = 'white'
    ctx.fillRect(0,0,WIDTH,HEIGHT)
    ctx.beginPath()
    ctx.arc(WIDTH/2,HEIGHT/2, WIDTH/2,0, Math.PI*2)
    ctx.fillStyle = 'magenta'
    ctx.fill()
    ctx.restore()
</script>

Preserving Chonky Pixels

We’ve seen that scaling up a canvas to handle high resolution is easy. But what if we need the opposite? Suppose we were making a game with retro-style pixel art then we want to see the chunky pixels. Unfortunately if we just scale it up the pixels get blurry, like this:

To fix this we need to tell the browser to not smooth the pixels. CSS already has a property for this, image-rendering: pixelated and it works for canvases too.

With that one css property it looks like this:

Note that if you draw scaled images inside the canvas they may still be smoothed. You can fix this by setting imageSmoothingDisabled:true on the drawing context

Big but not too big

Now that we have a retro game canvas, say a 160 x 120 pixel screen, we might want it to grow as big as possible to fill the extra space on page. We could use width:100% but the bottom might get cut off if the window is wide but short. Setting both width and height to 100% would make it fit but stretch and squish the pixels. Instead we want the canvas to be as big as possible while maintaining it’s aspect ratio, but not so big that any part of it is hidden. This turns out to be a common need for images as well, so there’s already a CSS property that does it: object-fit. With object-fit:contain the canvas will shrink to be completely contained within the styled size.

In this example a 160x120px canvas is stretched to 500x500.

canvas {
  width:500px;
  height:500px;
  image-rendering: pixelated;
  background-color: black;
}

If we set object-fit to contain then we get this, with black bars (the background of the canvas) filling the extra space.

And if we set object-fit to cover it will expand to fill all space, but possibly crop the edges off. Not good for our retro game but it could be useful for other cases where you don't know the rendered size until the page is loaded.

The object-fit - CSS: Cascading Style Sheets | MDN specs page describes all of the possible values. This css property is really for images, but remember: anything an image can do a canvas can do better!

Fill the Screen

So far we have used a canvas with a fixed size. We might scale the size up, but it has a definite size that doesn’t depend on the size of the browser window or any content next to the canvas. Sometimes we do want the canvas to adapt to its surroundings. Consider a drawing tool. It needs to resize when the user resizes the window. We could give the canvas a width and height of 100%, which would make it stretch to fill the available space, but, just like an image, that no longer preserves the aspect ratio. The pixels would get squished. Instead we need the canvas to change both its layout (css pixels) and its drawing surface (image pixels) to handle the current conditions. To do this we’ll need some code.

First we need to figure out how to get anything to fill the page. Let’s start with a div wrapping a canvas. Setting the width and height of the div to 100% does this:

Hmm. Not what we want. The problem is that the body element is special. It always takes up the full width of the window’s viewport, but the height depends on the content. What we want is for height to be exactly 100% of the viewport’s height. Enter CSS vh and vw units. Setting the div's size like this

div {
    width: 100vw;
    height: 100vh;
}

gives us this

Depending on your browser you might get some scroll bars. This just means there’s extra space we need to get rid of. Set the padding and margins to 0 for both the div and body, and set box-sizing to border-box to account for the 5px colored borders. Now we get this, and resizes properly with the window.

Now that the div is sizing properly, we just need to update the canvas itself when the window resized. Again we will use a little bit of code. On every size it will set the canvas size to the size of the div, minus the borders. Depending on your browser you might also need to set overflow: hidden; on the body element.

function calcCanvasSize() {
        // get the wrapper size
        const wrapper = document.getElementById('wrapper')
        let rect = wrapper.getBoundingClientRect()
        const can = document.getElementById('can')
        // set the canvas size to wrapper, minus the 5px borders
        can.width = rect.width-5*4
        can.height = rect.height-5*4
        const WIDTH = can.width
        const HEIGHT = can.height
        // redraw
        const ctx = can.getContext('2d')
        ctx.fillStyle = 'white'
        ctx.fillRect(0,0,WIDTH,HEIGHT)
        ctx.beginPath()
        ctx.arc(WIDTH/2,HEIGHT/2, WIDTH/2,0, Math.PI*2)
        ctx.fillStyle = 'magenta'
        ctx.fill()
    }
    calcCanvasSize()
    window.addEventListener('resize',calcCanvasSize)

And now we get a beautifully resizing canvas.

Conclusion

I hope today you've seen how easy it is to style the canvas to different uses. As browsers introduce more css properties for images, almost all of them will work for canvas elements. Go Canvas!

Code for everything in this blog is available on this git repo.