I have been using ChatGPT for 2-8 hours every day since the launch as a development tool. I’m interested in hearing what you think about it’s answers in your area of expertise?

For me, I found that it’s really good at generating new ideas, getting started on a project, common knowledge in a field, generating tutorials and other tasks early on in a development cycle.

I found that it is really bad at generating runnable code the more esoteric your coding task goes. I also found that many times the code or answers it generates sound plausible but when you run them they contain infinite loops or other major bugs. A lot of the time it will generate fake library names or algorithm names and fit them together in a way that looks good.

If you’re not already an expert in whatever you’re trying to do you can miss these things, for example I tried using Bazel for a multi language project without knowing Bazel and I only used ChatGPT with no docs for the project. I had a bug in my build that I spent 2-3 hours back and forth showing the logs and code to ChatGPT and it would always give me a plausible answer but the bug was still there. Finally I decided to read the Bazel docs and I was able to resolve it in 5-10 mins.

The overall take away for me so far is that this thing is going to vastly change how we work just like adding google search to your workflow did. However it also tells me that in the future expertise is what will be valued most, you can generate a million ideas but you need to be able to be able to tell which ones are good and bad ideas.