OpenAI / Github Copilot - It's not quite ready yet... but maybe it is for some. (Updated)

Update - It might be ready for some:

Like most things, you can find value if you lean into the strengths. I've continued to try and work with this and I admit there are times where it saves me some steps. For examples:

• In both flutter and Postgres, I found that it recognizes sequences and assists to finish entering them in.
• In classes, if I'm adding a function to convert the internal units of distance from meters to miles - Copilot quickly identified what I was trying to do, and wrote code to do the conversion including null checks. 
• Writing out a comment for refactoring and having it modify existing code with the refactoring in place

In these cases, it was actually about 95% correct (meaning no additional changes were required on my part). I'm not yet convinced we are seeing a value worth $10 / mo yet, but it's probably worth half that. I'll document more as I use it further. 

Original Post:

OpenAI / chatGPT / Github Copilot:

Having kept up with the chatGPT craze, I was excited to try out Github Copilot. It promises many great benefits, including:

• Converting comments to code
• Creating unit tests
• Writing SQL queries

Having used OpenAI before via chatGPT for code generation/review, I knew it was capable of much more. I suppose to be fair, I did not test the touted features (list of 3 above) of this platform specifically - at least not line-by-line like their examples, but I feel like my expectations were reasonable for a service like this.

What I've tried it with so far: 

• Flutter / dart development including implementing the supabase library
• Programming C++ for Unreal, specifically writing a custom DMX Modulator for RGB to HSI conversion
• Converting SQL queries for Postgres

Issues:

• Many suggestions were invalid. This is likely due to that fact that the training data for this model cuts off back in 2021. Anything produced since then is more or less unknown. This was evident with the outdated suggestions on Unreal and outdated library suggestions for supabase. 
• The SQL conversions were only partially correct. It was unable to do the full conversion.
• Before Copilot, I used chatGPT to seek assistance with some issues, I was met with conflicting results. The first was the outdated code problem, but the second was that when I would tell it the code was wrong, it would apologize, admit that a correction was necessary due to a newer version and then print new code which usually ends up being a mash-up of old library calls and a few lines updated for the new library. When I asked what it knew about the new library (citing the same reference it gave before) it said it knew nothing of anything from after the cut-off date. The outdated code issue was also present in the IDE with suggestions the Copilot gives. Many of these were incorrect.
• Lack of customizations / rules. AFAIK there is no interface with the service from my IDE (Visual Studio Code) at this time. This means you can't customize how and when copilot does it's thing, other than completely enabling or disabling the service. 

Conclusion:

I imagine a lot of this is actually fairly well-known already by those building the service or having been using the service for much time. The only real benefit (maybe) to this point has been the auto completion a couple times. Certainly not worth the $10 / mo price tag right now. I would love to have a UI that let's me direct the service in how I want it's help or even an icon that pops up that I can click on to see the suggested improvements / optimizations.  

These should be a standard part of the service:

• Reduce bugs
• Bring knowledge base up-to-date
• Add/Improve UI in the IDE
• Add knowledge for the code-base as a whole
• Identify programming patterns in-use, and be consistent with them with suggestions (I actually do not know if this is available, but since it doesn't seem to know the other files in your project, I assume it is not)

Once these are implemented, I think this service would be amazing at $10 / mo.

A final word about ethics. Some people have criticized these kinds of AI services (i.e. Midjourney, OpenAI, etc.) because they produce value based on the analysis of existing works (i.e. art, public code repositories). Many of these works are protected or licensed, however what is being produced is derivative which to my knowledge, generally has certain allowances legally (I'm not a lawyer). The question I come back to is: Are original works, actually original works, or simply derivations of other inspiring things? If I create something artistic after visiting a museum, writing a song after hearing another inspirational song, or write a story after reading an inspirational story, am I really creating original art? There is a reason there is an entire field of law dedicated to these questions. But the bottom line seems to be, if you don't feel right about it, don't support it.

As a bonus, I ran the content above through chatGPT to "make it better", lol:

As someone who has been following the chatGPT craze, I was excited to try out Github Copilot and see what it could offer. Its promise of converting comments to code, creating unit tests, and writing SQL queries seemed like a game-changer.

However, my experience with Copilot has been mixed. While I've used OpenAI before via chatGPT for code generation and review, I found Copilot's suggestions to be sometimes invalid and outdated. This is likely due to the fact that the model's training data only goes up to 2021, leaving it unable to provide accurate suggestions for newer libraries and frameworks.

For instance, I encountered issues with outdated library suggestions for supabase and Unreal, and SQL conversions were only partially correct. When I sought assistance with some issues using chatGPT, I received conflicting results. While it would apologize and admit that corrections were necessary due to a newer version, it was limited by the cut-off date of its training data and could not provide knowledge beyond that point.

Another issue was the lack of customization options, as there is currently no way to customize when and how Copilot functions within my IDE (Visual Studio Code). This means that I can only enable or disable the service entirely.

In my opinion, Copilot needs to reduce bugs and bring its knowledge base up-to-date. It also needs to add an interface within the IDE to allow for customization, provide suggestions based on programming patterns, and identify consistent coding practices. If these improvements were made, I believe the service would be worth its $10/month price tag.

On the topic of ethics, some have criticized these types of AI services for producing value based on existing works, including licensed or protected code. While derivative works may have certain legal allowances, it's important to consider whether original works truly exist or are simply inspired by existing ideas. Ultimately, it's up to individuals to decide whether they feel comfortable using such services.

In summary, while Copilot has its benefits, it still has room for improvement. With updates to reduce bugs, improve the knowledge base, and provide customization options, it could be a game-changer for developers. As for ethics, it's a complex issue, and individuals must make their own decisions based on their personal values.