Are Tech Companies Forcing AI Features Nobody Asked For? | Utilizing AI Episode 46

We're surrounded by new AI-powered devices, many of which seem to have been designed without much thought given to their impact on society. This episode of "Utilizing AI" features Olivier Blanchard and Guy Currier of the Futurum Group discussing the way companies are bringing AI to market. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence and part of the Futurum Podcast Network.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Stephen Foskett, president of the TechFieldDay business unit here at the Futurum Group. Before we dive into this discussion, let's meet who's on the panel today.

Olivier? I'm Olivier Blanchard, research director with the Futurum Group, and I focus on the intelligent edge and AI devices. Hi, I'm Guy Currier, and I'm also a research director at Futurum Group, as well as one of the directors at the Futurum Media division of Futurum.

And that means that we are eminently qualified to discuss today's topic since it affects us all, but of course, it affects everybody. And the discussion that we're focused on today really is sort of the intersection of this Silicon Valley culture of aggressively moving forward with new features and the world that is now finding ourselves adopting these things increasingly. In previous podcasts, we've talked about the spread of AI generally into everyday lives.

We've talked about AI in mobile devices and automobiles. We've talked about AI in the home. With so many new AI-powered features being rolled out all the time, it's fair to say, where are these features coming from?

Are they being driven by the needs of the marketers, by the needs of the consumers, or just sort of on autopilot, trying to find a market for AI technology? So when we think about this, are these features designed for good or for evil? Where do you stand on that, Olivier?

Yeah, so I think the issue here is that AI obviously is a tremendous toolkit that allows us to do a lot of very interesting things, solve a lot of problems, improve systems that need improving. But the flip side of that coin is that it's also kind of dangerous, and it can be used for nefarious reasons. It can be used improperly, and it can ultimately cause more problems than it solves.

And so we find ourselves at this weird intersection with this new technology that's still kind of messy, trying to find its way, trying to find its purpose, trying to justify the tremendous historic sort of unprecedented investments that we're seeing into AI and sort of form, create some kind of return on investment for the companies that are backing this push for AI. But what gets lost, I think, in a lot of this is beyond the investment to ROI pipeline and all of the things that AI could do, is what should we use AI for? And also what shouldn't we use AI for?

And one of the ways that it manifests itself for us, the user of AI, dare I say consumers, but also business users, but as human beings who interact with technology on a daily basis, is how it impacts our privacy, how it might impact our freedoms, how it impacts our quality of life. And I feel like the math on this is still an open question, right? But a lot of the features that I'm seeing sort of popping up every few months in devices specifically, so smartphones, smartwatches, smart glasses or augmented glasses, PCs, tend to be things that nobody asked for, right?

There's a small subsection of experiences and features that we've been asking for for a long time, right? Voice recognition, wake up on commands, the facial recognition to unlock your device, things like that that sort of reduce friction are really great. But we're also seeing sort of an increase in the number of features that nobody asked for.

Like devices that are always listening, devices that are always recording, devices that track us and follow our faces around a room and interpret what we do. We're starting to see cameras inside of dishwashers, not because of efficiency reasons, but presumably because they want to track when and how you load your dishwasher and what goes into the dishwasher and what doesn't. Which in turn is data that can be sold to somebody who might be trying to understand your behaviors, your habits, the timing of what you do, what you fed your family that week, et cetera.

And so I think that tension between what we would like AI to do and the things that we would like to opt into, as opposed to what we don't want AI to do and the things that we may want to opt out of and may not be able to, that tension is at the center of pretty much every new discussion that we have about emerging features in AI. Well, I think that's a terrific perspective, Olivier, and I think there's a question in your, let's say, soliloquy about the situation we're in, the question that you are asking without asking, which is: how the heck do these things come into existence in the first place? Why are we getting the features we didn't ask for, as you put it?

And I think that's actually a really critical question when it comes to AI, because you know where they come from? Well, they come from human beings. But they also come from human beings using AI.

Say you're a product manager at Apple, and you're thinking about, "Oh, AI's coming, and I don't know what my job's going to look like and all these other sort of things. '" What am I going to do? I'm going to say to AI, "Here's my product.

Here's my corner of the product. " Or the dishwasher product manager or what have you. We are, as knowledge workers at least, hard-pressed to find any corner of anything that we do that we can't at least use a quick AI chat for.

And I'm mindful while I'm doing this, and I'm mindful while I'm speaking on podcasts or whatever, to remind myself and to remind everybody that AI is simulative. It is trained to spit out something that looks original, creative, and new, and yet it has been trained based on original and synthetic inputs that are based, in turn, on the current human corpus. So what we're going to talk about in terms of new ideas, they look like new ideas.

They may not be new ideas. They look like great ideas. They may not be great ideas.

This is the trap that we are in. This is this cycle and circle of Ouroboros, the snake eating its own tail, and that is a coming, gosh, I don't want to say singularity because of the meaning of that, but that is a coming inflection point, let's say, of its own. I would be a little more charitable than saying that AI is designing these features for itself.

But I actually share a little bit of cynicism here with you, Guy, because, well, first off, the dishwasher hits home because we bought a new dishwasher, and we don't use half the features on half the appliances we have, but in my dishwasher, I kid you not, you cannot use the delay start feature, which is just a basic feature, without the app. And so you actually have to get out your smartphone in order to do the delay start. It's like you have eight buttons, and none of them does the thing that is the one thing that I actually might want to do.

But more to the point on a lot of these features, I actually feel like one of the biggest issues is that we have AI technology, generative AI, and large language models is essentially a solution in search of a problem at this point, in that it's a solution in search of a market. " But more to the point, even before that, I think what they're saying is What can we do to incorporate large language models, generative AI, et cetera, in this angle of the world just to kind of justify, it's like self-justification. I have this technology.

What could I do with it? I don't know. Maybe I'll, whatever.

And I think that that's one reason that so many of these features are appearing, and I want to get back to one of the things that Olivier said there, that nobody ever asked for. Because I think that that's the important thing is that if you look at a lot of these features, on the one hand, you could say, yes, these are serving a need in the market, but are they? Or are they really sort of solutions in search of a problem?

Yesterday on "Security Boulevard," they were talking about the Apple iOS, watchOS feature where it's always listening, and you can ask for a summary of the last few seconds of the conversation on demand if needed. Now, Apple has a good reputation, as Fernando pointed out in that discussion, for privacy protection and so on. But the thing that I would say is, that's an example of a great feature that nobody ever asked for.

" Right? And maybe that's a good idea. No user was clamoring for an all-glass, no buttons phone until Apple came out with the iPhone, and then everybody wanted it.

But I don't think this falls into that category. I think this is a minor, not even a B, but like a C kind of feature that might not even make the keynote, and yet it runs the risk of really polluting Apple's privacy reputation. For what?

For nothing. And it's kind of the same with a lot of these AI-based features that we're seeing rolled out there in, I mean, look at the incredible backlash against Flock cameras. A lot of the stuff that that system does is not something that law enforcement wish they had.

" And boom, there you go. What do you think, Olivier? Yeah, no, exactly.

That's exactly the problem. And so I'm going to go back to my old career in marketing and product development, where I was hired by this company, and it was ironically enough, to work on a dishwashing product, which has nothing to do with the dishwasher that I was talking about. But they'd had the same product for decades and decades, and it was well-loved, and they did fairly well with it, but they were thinking about doing a redesign, right?

And so I came in, I knew nothing about this. The first thing I started to do was ask some questions. And I went out in the field and talked to people who used it and said, "Hey, what do you like about it?

What do you not like about it? " And that type of market research, which took a little bit of time, took a little bit of diligence, or required you to interact with people outside of an office, to go out in the field, actually experience it yourself, led to some really interesting insights. And based on those, over time, we developed a completely different version of that product that was updated based on all the feedback that we got.

It was faster to install, it was lighter, it was cheaper to manufacture, it performed better, it didn't burn people's hands. It did all these things that made everybody happy, the installers, the plumbers, the users, the restaurants, everybody. And that sort of model, that diligence of knowing how to ask the right questions, knowing where to go to ask the questions, which generally is just talking to the users, interpreting that feedback, iterating solutions until you find the right mix of design and engineering and sort of manufacturing cost management, that's how you get to the right product.

And we did this in the timeframe that it would've normally taken for a regular refresh cycle or a regular design cycle. So it's not like it took twice as long as it would have otherwise. That is the sort of methodology which seems antiquated today because it didn't include AI.

It didn't include just sitting around tables having discussions about what we should do, what we could do. It was literally going out and talking to people. And I feel like technology companies now, and it's not to say they don't do market research, but I feel like the questions are rhetorical.

When they do reach out to the public, to users, it's to test features that they've already decided they were going to implement to see how they're going to land. As opposed to asking, "Hey, what would you like to see in the next smartphone? " And that's clearly not happening, and I think the root of the problem is that technology companies have either stopped asking those questions or have forgotten how to ask them and who to ask them to.

Either that or they're asking the questions and they're not listening to the answers. But whatever the mix of that problem is, that's where we are today, where technology companies, product teams are basically throwing noodles at a wall to see what will stick and just coming up with ideas that are disconnected from the needs of the users, in order to push data collection, and sort of data analysis through the devices out to their partners, which doesn't serve their user and ultimately lowers the value of each new generation of technology, lowers the value of AI, and I think negatively impacts the drive for demand for these new features. And so it's just sort of like, to go back to the old words, it's the snake eating its own tail and getting smaller and smaller and smaller, trying to extract more and more data and value out of the users instead of trying to create more value for them to drive more demand.

And so the equation's backwards and it's reductive, and I think that's why we're seeing a backlash against AI, why we're seeing so much of AI sort of turning into a surveillance state product as opposed to features that should improve our lives. And it's a mistake and it's a dangerous path. Well, Steven, make sure you mail Olivier that John Birch Society application because- Oh, no.

No ... I don't think we're really going back. I have neither worked in the dishwasher industry nor bought a dishwasher recently, but I can say this much, which is that, that kind of classic product marketing discipline, product management and product marketing discipline Olivier was talking about, really in a lot of ways, never took hold in cloud or B2B, enterprise or business product development, mostly because those sectors are so heavily sales-driven that rather than doing that kind of research, I mean, it's not that they never did it, but rather than doing that kind of research, there was a lot more to be made from responding to salespeople, especially ones with big deals on the table.

I think it's still alive and well, to a certain degree in consumer, but Olivier, I think your fundamental point is really sound because that's going to get, if not already, seriously infected by AI as well. Everything you described from your dishwasher days, ooh, I got to put that on a T-shirt. Everything you described from your dishwasher days, Olivier, can be enhanced very well with the use of AI.

I'm going to repeat, I think I'm just going to say this on every podcast till the end of time. The primary benefit of the use of AI is quality. Quality and a certain reliability or dependability of execution because you can get AI to start stuff that's hard to start.

I really don't think it's productivity. I think productivity may be a secondary benefit. But it's quality because you can do deeper research, you can very easily and quickly get lots, a whole host of ideas.

If 90% of them are wrong, 10% of them are right, maybe half of those 10% you wouldn't have thought of yourself if you're using it in that kind of a way. I'm continuously struck by how little understanding there still is of what AI is, how it works, and that small bit of value you can get out of it. Actually, well, it's great value out of maybe 10% of what the vendors and the public are saying or all your peers are saying you can use it for.

Every time now that I hear somebody say, "Oh yeah, I vibe coded X," or, "I used this agent Y to do this thing that takes you two weeks to do. " I always put an asterisk on it in my head because I think what most people are seeing is the great promise here, and not necessarily, that's not they're being dishonest, it's just your experience of suddenly being able to do something so much quicker Overwhelms the senses, especially when, again, the systems are designed to fool you. That is their design point, is to appear true, correct, creative, awesome, whatever.

Put that in the hands of somebody who does not know what they're doing. That's where a whole lot of the risk and problems and cost builds up. Yeah.

" But yet, I actually agree with you 100%. You're right that the primary use case, the real use case for a lot of this technology is to improve our ability to do things and to take in more data, to process more data, to take in more perspectives. That's how I'm using it right now is as a research assistant.

I have Claude connected to many internal systems, and I use it almost exclusively to help me get some context around this thing that I'm thinking about by searching Slack and email and Salesforce and so on, and letting me know the things that have gone out of my head. I'm not using it to write and so on. But the reason that I reacted so strongly initially to your point is because that is not how it's being used.

It is being used to replace thought and to replace work. But it's terrible at that. And you look at all these situations and it's...

But I don't want to lose the thread here of this conversation, which is about product design and products. And I think that that's actually a risk there as well. Because if you think about many of these AI-powered features, if they are a way to bring in more data and to do more processing to help us do our daily lives, then I think they're good.

And the, I'm not going to say the name, but the S lady that lives inside your iOS phone now can search calendars and email and private data in a private way and surface that for you. I think that's actually a really nice AI-powered feature. But the features in the same operating system that can, for example, fill in extra space around a photo that you cropped improperly are perilous features.

Because so far, when I've experimented with these things and I've tried Image Playground, or I've tried doing the cropping thing or doing a lot of the generative things, what I've found is that it looks like you said, Guy, it looks at first blush, wow, that's cool. But when you look at it and you're like, wait a second, why did it decide that this concert that I went to see was outdoors instead of indoors? That doesn't make any sense, but that's what it did to my photo.

And now that's my photo. What happened here? So I think that we kind of get back to these AI-powered features that, again, I feel like they are solutions in search of a problem.

There's somebody trying to, to Olivier's point, throw some spaghetti against the wall and see if that's what consumers want. Maybe they want this, maybe they want that. And if I was Apple, maybe that would be a good thing to put in iOS because frankly, they're behind on AI.

They don't know really what it could do and what people are going to gravitate towards, and I guess we'll find out. Yeah. So here's a genius idea for Apple.

And I think it illustrates the issue that was mentioned on the podcast yesterday about the watch, right? And this sort of always-on listening feature that nobody necessarily asked for. It's a really good technology.

It's a bad application of it. And the question at the root of every single design choice and every single feature should be why? What's the purpose of this, right?

So if nobody asked for it, that's okay. Maybe we just thought of something that they would want to have before they did, just like a no-keyboard touch screen, and Apple did that. And occasionally, you see this.

But there are also problems that are sort of obvious if you're aware or if you care enough to go look for them. So a solution in search of a problem to me in tech is just criminal. Let me give you statistics real quick.

Over 20 million women in the United States every year are the victims of domestic abuse. What if that exact feature that is on the watch were used with a safe word of the choosing of the user, in this particular instance, in this context, a woman. She gets the watch, she activates the feature, determines what the safe word is.

When the safe word is uttered by her voice, the watch starts recording, right? And now you have a device that knows to record when something violent or domestic abuse or something else is about to happen or is happening. And now you have evidence, and there's more protection, and maybe it does other things in the background, like alert authorities or call her friends, right?

A sequence of features that would help make women safer and feel safer and do something about making themselves safe when there's no one else around to protect them. That is purpose, right? That is a purposeful application of that AI feature.

As opposed to, hey, we have this AI feature. We've never thought about any reason why anyone would have to use this. Let's throw it in there without purpose, right?

And this is what I'm talking about, is just even if product teams are not incentivized to go out and talk to users, it's not difficult to go find a problem in the world around you and use the technology that you have at your disposal to solve that problem. And if it's used by marketing teams in a cunning way to sell more product, so be it. I don't have a problem with that, so long as it solves a real problem for users.

And I feel like that would be a really good marketing tool as well for Apple in terms of cementing not only its privacy reputation, but also its value to society, right? And so it's a win-win for everyone, except for domestic abusers, which is fine. But that's the kind of thing that I would like to see more in the application of AI, whether it's in devices or elsewhere.

And I'm not seeing enough of it. Live translation is great. There's a lot of vision impaired technologies that allow your smart glasses and your phone when you point it at something to help someone who might not be able to read or see something very well to understand their surroundings.

Those applications are wonderful. Most of the ones that we're seeing are just surveillance slop, and they bring no value to the equation. I'd like to see that reversed.

And it's just a matter of choosing to have purpose and to give purpose to the technology, as opposed to just throwing it out there for no reason at all. That's a really great point, Olivier. And it's really a crying shame because that's so close to what they did.

Yeah. And it could've been a highlight and a real selling feature. For example, your feature is actually very close to one that Tesla has, but it's turned off by default.

Mm-hmm. So get this, so in your Tesla, it's got all those cameras and stuff. There's an option that every time you honk the horn, it saves the previous 30 seconds of camera from all the cameras and the next, I think, 30 seconds or one minute of camera or something like that to a separate spot, and then that doesn't get overwritten or deleted.

Because think about it, whenever you honk your horn, that's when something happened that you might want a quote, dash cam video of. But not only do they not talk about this feature in advertising, it's off by default. And so something that could actually be a really cool feature- Yeah ...

and actually a feature that helped us when we got rear-ended in the car, is something that is not advertised. And like you said, having a watch be able to do that, it's not necessarily a feature. Hopefully, it's a feature that wouldn't get used a lot.

But it's a feature that would get a lot of cred for the company. Yeah. Just like when Apple did the satellite connections for the phone, and they emphasized the fact that it could help hikers that get lost in the wilderness.

Well, that's not a high- Right ... numeric- It's not 20 million a year, right ... but it actually did happen, and it actually was used, and it actually saved people's lives, and it makes people feel good about the company and good about the product.

Yeah. And similarly, the feature you described, it's the same kind of thing. And it's too bad that marketers aren't thinking about how can we use this technology in a way that preserves privacy and helps people and is really a signature instead of sort of, oh, and by the way, we're spying on everything you do for marketing purposes.

Correct. That's it. I don't know.

I think we might be, or you guys, since I'm just weighing in now, might be a little unfair here towards- Never. That would never happen ... towards Apple.

We need to distinguish between what's conventionally called platform versus product or service, because Apple really is in the platform business. It can get confused because they produce the apps, products, services on top of those platforms. This capability that you're describing, Olivier, it's like an application of the platform capability to listen and record that the watch has.

So I don't see why you can't go and create that application that listens for a word, a safe word, in the voice of the user using the platform services provided by the watch. And by the same token, if Apple is producing this platform where that application is possible, we can also imagine, although we shouldn't say here, all sorts of nefarious applications of the same thing. And I think that's where, to sort of bring this back home, where this whole admixture of AI and the use of AI in the creation of these things, incorporated into all these things, creation of these platforms, the creation of their capabilities, also the incorporation of AI into those application services, products, and so forth, its pervasiveness.

And yet it's this weird dichotomy between what it seems to be doing and how it's actually providing value. That's where all of this gets, at best, chaotic and uncertain, and at worst, downright dangerous. Right.

No, I agree. We opened with this sort of flip side of the coin, right? The value versus the risk.

And any technology can be used to nefarious ends. But I do feel that, especially with Apple, and it's not just Apple, right? We're talking about Apple because Apple was mentioned yesterday.

It could be Samsung, it could be Pixel, it could be anybody. But I feel that especially with Apple, because they have the full stack, right? They're coming up with their own silicon, designing it.

They have this walled garden and this whole stack of technology where they are the designer and the implementer. These features do matter. And when you're presenting certain features as sort of endemic to your new platforms, your new products, you do have a responsibility to kind of like, you can't take credit for it and then also just be like, "Hey, wait, we're just Apple.

We're just coming up with chips and designs," right? But I feel like this sense of we're just going to throw features out there and hope that they work could be better if we took this technology and said, okay, there's a problem with domestic abuse. There's a problem with bullying.

The smartwatch could help with both of those problems. It's not perfect. It's not a complete solution, but it can help in those cases.

And it could impact positively tens of millions of people, whether it's women or children or whoever else uses the technology for their own safety. And at that point, it's not as intrusive and as sort of like, nobody asked for this anymore. It's sort of like, oh, I know exactly when and where I would use this, or I'm going to get this for my sister, or I'm going to get this for my nephew.

And it has the potential to bring more value to the product, to the technology, to the experience, and to the world. And it does, even if it doesn't solve the problem, it at least tries to address the problem. And I think that's just a better tack for tech companies to take, whether they're Apple or anybody else, than just sort of throwing a bunch of features nobody asked for that feel increasingly intrusive, and tying that to the value of devices that are already expensive enough to replace without the added stress of wondering if they're going to be spying on you when you're sleeping.

So I think it's just a better direction, not a perfect solution. Yeah. And it really is a better direction.

I hope that marketers are thinking about this and listening to this because I will just say that in my conversations with people, there's a lot of concern about AI. There's a lot of concern about technology in our lives. And I feel like a company that can go out there and firmly and consistently say, we're on your side on this discussion, and we've thought about this, and we've created, I think that would be a great marketing differentiator.

And frankly, maybe it's not going to be Apple. Maybe it is going to end up being Google or Samsung or somebody else who finally realizes, wait, that's a good marketing approach as well as good for people. And I would love to see that happen.

So thank you so much. Really a lot of fun. I love having you on here every week.

Thank you so much for joining us on this episode. Before we hop off, though, Olivier, where can people connect with you and continue this conversation? Well, The Futurum website, obviously.

Also, I'm still on X. I don't love it, but I'm there. Occasionally on LinkedIn, and then at industry events as well.

So if you see that I'm tweeting or posting about an industry event live, feel free to approach me there. com. You'll see my writings and analysis there.

com/in/guycurrier. social. And as for me, yes, I still have an X account, but I'm more commonly posting on LinkedIn, Bluesky, and even the, yes, the Mastodons, the Fediverse.

So you can find me as sfossum on most of those sites. " If you enjoyed this discussion, please do subscribe on YouTube or in your favorite podcast application and consider giving us a rating and a review. This podcast was brought to you by the analysts and experts from the Futurum Group, where insights meet AI.

tv app. Thanks for listening, and we'll see you next week.