Was using my SO's laptop, I had been talking (not searching, or otherwise typing) about some VPN solutions for my homelab, and had the curiosity to use the new big copilot button and ask what it can do. The beginning of this context was actually me asking if it can turn off my computer for me (it cannot) and I ask this.
Very unnerved, I hate to be so paranoid to think that it actually picked up on the context of me talking, but again: SO's laptop, so none of my technical search history to pull off of.
I think AI is humanized and otherwise designed so that people will feel encouraged to give private data to it. The Kagi Corporation wrote about this in their manifesto. In reality, giving your data to open AI is just as unsafe as typing in a personal search query into Google or Bing. But by changing the context, it feels like you're talking to a friend or a person you met at a bus stop.
AI Bros always say "it's just a tool" as a sort of thought terminating cliche (note: this wasn't intended to be a dig at your comment). Guns are a tool too. I wouldn't want the richest corporations in the United States to personally own the most powerful missile systems, and in terms of AI, that's kind of where we are.
Brilliant! I love your closing point about tools and missile systems