This week was very interesting as well. I worked with some familiar tools like postman and ChatGPT and some not so familiar ones like the python mysql library.
GPT was able to do some pretty amazing things with little guidance. One interesting that happened this week though was that it lost context in our conversation. I’d seen this happen in multi-turn conversation evaluative testing at work, but I’d never seen it happen in a live model. We were talking about the CRUD operations and it was generating near perfect code when suddenly I asked it for the next one and it completely switched formats on me. Specifically, it forgot that we were receiving API variables as headers in the API call and instead expected them to be in an external doc that was opened and stored in a dictionary. It was very interesting to see this. Eventually, after a few extra prompts reminding it what we had been doing, it recalled the API part of things, but still did not remember a few other details like the exponential_backoff wrapper we’ve added to all of our API endpoints. That one issue aside, thorough testing has shown that GPT produced near perfect code for all of my use cases.
Leave a Reply