fun read, its weird interacting with chatgpt around markdown sometimes.
it formats its own stuff with markdown, so if i ask it for markdown but dont explicitly specify a downloadable file, it will produce valid markdown up to where it conflicts with its own markdown, and then it gets choppy and chunked.
its an issue of my prompting is what im sure some customer service rep would be told to tell me :p because theres money to be made in courses for prompting skills perhaps, idk. (cynical view).
sure is enjoyable to struggle together with the AI to format its responses correctly :'D
Could this result in edge cases with [ where due to some misformatting or intentional syntax that looks like the start of a markdown link, the entire response is hidden from the user?
(This comment when subjected to this processing could look like: "Could this result in edge cases with ")
I do something like this too because links in emails are insanely long. It's worse in marketing emails. So I shorten the links to save on tokens and expand them again when I get the response back from the LLM.
Is there a general solution to this problem? I assume you can only start buffering tokens once you see a construct, for which there are continuations, that once completed, would lead to the previous text being rendered differently. Of course you don't want to keep buffering for too long, since this would defeat the purpose of streaming. And you never know if the potential construct will actually be generated. Also, the solution probably has to be more context sensitive. For example, within code blocks, you'll never want to render links for []() constructs.
EDIT: One library I found is https://github.com/thetarnav/streaming-markdown which seems to combine incremental parsing with optimistic rendering, which works good enough in practice, I guess.