I’m often explaining a system to another engineer, and the suggestion comes up: We should document this in a system diagram for the future! But online whiteboarding solutions are always high-friction enough for the task to get backlogged forever.
What if we could draw the system diagram during this conversation, by using the conversation as the input to describe the system? We will need:
This is a large project. As such, it makes sense to break it up into some small deliverable milestones. The first: use the microphone to transcribe onto the tldraw canvas. This stage helps us test Whisper’s transcription quality and the tldraw editor API.
Whisper worked great for a couple short sentences, but I noticed that it didn’t work so well for complex trains of thought with typical “service names”. To fix this, I might extend the transcription feature by piping it through GPT4 to fix misspellings and substitute in common technical phrases.
A simple transcription of a long sentence doesn’t fit right on the canvas. (See demo for an example.) It’s likely that we want to split this up into N-character chunks, demarcated by word boundaries.
Luckily, we only need to pass one function, editor.createShapes, to our function-calling parameter for the next demo.
https://twitter.com/AarohMankad/status/1726738749894406640