Testing Apple Intelligence's Clean Up Tool
These two photos are the same photo.
Don’t believe me? I know, I know. At a glance, you might think “one is a desk with things another is and empty desk there’s no way these are the same photo”. But then you might give into curiosity and start to notice a few weird things about the second. Some weird shadows and some very straight lines, almost too straight, in the wood grain.
Well the answer is that I took the first photo, and used Apple Intelligence to clean up the desk (pun not intended) with the Clean Up feature within the Photos app. Read on to know a little bit more about this new functionality.
Apple Intelligence
As I see it, Apple Intelligence is Apple’s machine learning (ML). With the advancement of artificial intelligence (AI) and large language models (LLMs), Apple had been resistant to using the acronym “AI” and stuck to machine learning whenever they had any feature powered by some sort of algorithm resembling AI.
In a very Apple move, Apple launched “Apple Intelligence”, and rebranded the AI we all know to something new.
Yes, Apple, we all saw what you did there.
Now with Apple Intelligence, they can use the acronym more freely. And although they released a new phone built for AI, no Apple Intelligence features were seen on the iPhone 16 when it hit the stores back in September 2024.
In October with iOS 18.1 we started to get a few features from Apple Intelligence and I, like many others, updated my Apple gadgets, the iPhone and iPad, to iOS 18.1 in order to use the first few features from Apple Intelligence.
Testing out AI in Photos
Talking to a friend we noticed we did not use the features yet, so I went into learning mode to figure out what the features were exactly and see how they could fit in my life.
Out of all of them, the one that caught my eye was the “Clean Up” feature in the Photos app.
Sure, removing unwanted objects and even people from photos is nothing new, many android phones have had this functionality for a while now, and traditional photo editing software are great at it, but this is not a blog post on the merits of each operating system or editing software.
If you know me, you know I’m always curious about the limitations of tools and I wanted to see how far I could push this one… My goal was to “remove” one object at a time, save a new photo with the removed object, and continue on until I had a clean desk top.
iPhone or iPad?
For the curious ones here are my specs: I’m using the iPhone 16 Pro Max with 256GB and the iPad Pro 13” M4 256GB launched in 2024.
Starting on the iPhone
I started on my phone, laid out the things on the desk, took the photo, and sat down on the couch with my phone in hand to remove the first object.
Can you spot the difference? Look at the bottom right corner. It felt like magic when after a simple circle the thing just vanished.
I saved the edit, duplicated the photo and went on to the second object: it was time to remove my tiny army knife on the bottom left. It took a little while longer than the first one for it to disappear but it go way. But it left a “shadow”, so I had to also circle the shadow and wait a little while for it also vanish.
After duplicating the new photo, the iPhone was officially a bit hot and started freeze up in the editing process, so I moved on to the iPad to continue with the fun.
Two notes here:
- First, when the iPhone 16 launched, Apple said that this iPhone was built for AI, but this tiny test I did suggest that the iPhone 17 might be better at it,
- second, that duplicating photos also saves up all the edits from the past photo as an edit, so you can revert a duplicated photo back to the original for example.
With these in mind I sympathize with the folks with the iPhone 15 that will want to use this feature extensively and that I hope Apple releases updates in the future to address this and some other quirky behaviors you’ll see below.
Continuing on to the iPad
All things considered, it was super fun to simply circle the object you want to remove and wait a few seconds and it go poof and just vanish.
After “vanishing” an object sometimes I also had to address some dark spots left over, I just had to circle the area again to get rid of any leftover dark spots (like shown in the video below).
As I expected, the iPad performed better than the iPhone, I didn’t notice it freezing up although the tool responsiveness got slower the further I went.
And as far as temperature goes, the iPad only got warm, but whenever I got closer to the screen I could feel the warmth emanating from the iPad.
Then Some Weirdness Happened
After all objects but one were individually removed, I had a last challenge: to remove the biggest item of the whole photo. The objective was to go from the left to the right.
But what looked like a simple thing after everything I already done turned out to be the moment when the weird things started to happen. I wonder… Does Apple Intelligence, like any other AI models, hallucinate?
Hallucination is a known behavior of machine learning, especially in LLMs, and in my opinion one of the most dangerous ones. An hallucination in ML happens when a model produce a coherent response but that is in fact incorrect.
Since I don’t know if we can qualify if Apple Intelligence was having a hallucination here, especially since we are not interacting textually, I’ll call everything in the realm of unexpected results a “weird behavior”.
Just a blur for you
The first two attempts to remove the book did not go well. The clean up tool stopped removing things all together and started to “blur” things instead of removing objects.
After a few failed attempts (one of them you can see in this video below), I finally got to it to do what I wanted to do and the book was removed. Mind you that it required some extra work on my end since circling was not doing the job.
I wish there was a way for me to tell the tool that it did the wrong thing and to try again, like many text-based LLMs, but there’s no such way. Here you have to hope the feature understands what you want it to do. ¯\_(ツ)_/¯
There used to be something here
Still on the weird behaviour spectrum, when I tried to paint over the part where a lot of the editing was done before, it showed the “crop out” of the object that was removed previously.
Eventually it worked out and I managed to get the photos you saw in the beginning.
Cleaning Up is Tiresome, Even for AI
One other tidbit of information was this was a very battery-intensive task. Editing itself didn’t take long, each object removal took only a few seconds.
However, I could see my battery percentage go down very steadily. I believe since I started the editing process in the iPad until I finished editing, this feature consumed about 20% of the battery available.
I’ve done a few intensive tasks in the iPad before like editing videos and digital drawing with multiple layering, but nothing even got close to this.
I also noticed that it started to drop faster the more I accumulated edits in a photo. So I definitely would not recommend using this while relying solely on battery and unable to charge your devices. If not maybe buy a powerbank (this is one I like).
The end result
After getting all my photos there was only one tiny thing left to do this tiny video from “only one photo”.
Could I just take the bunch of photos in half the time it took me to do all the “cleaning up”? Sure I could. But what is the fun in that?
Recap
If you read until here you may be wondering why would this be useful at all? Well imagine you have a lettuce stuck in your teeth but otherwise the photo is perfect, now you can simply remove it.
A friend pointed out to me that you could also remove people you don’t like anymore from an old photo or someone photobombing your cool vacation photo too.
In short the new feature is useful but it is important to be aware of its limitations and quirks as well as the possible battery drain.
Thanks for reading.
This post contains affiliated links.
