The new iPhone operating system incorporates an optical character recognition function. With it, you can extract and translate on the fly any text displayed on an image or photo.
The integration of an optical character recognition module is nothing new on smartphones. For several years now, Google has offered such a function in its Photos app, which includes the Lens optical recognition module. With iOS 15, Apple is therefore catching up a little and incorporating a similar function.
Live Text, that’s its name, can search, extract, and even translate, any text displayed in images. To benefit from it, however, you must have an iPhone equipped with an A12 Bionic processor or newer, that is to say from the iPhone XR.
In addition, this module can be used from several applications in iOS. Apple has integrated it into its Photos app, in the Live Camera preview, in the Screenshot feature, as well as in Safari. Here’s how to use it to extract text from an image for use in other applications.
1. Recover text from Camera Roll
All the images and photos already stored on your iPhone Camera Roll can be used to extract text. To do this, open Photos and view one of the images stored on your iPhone.
If text is present on the image, iOS automatically detects it and displays an analysis button at the bottom right of the image. Tap on it to start analyzing the image. When iOS has detected all of the text present, it is highlighted with a very light highlighting.
At this point, the text can be selected. Keep your finger pressed for a moment on one of the words to activate the selection and adjust it by pulling the two handles present at the ends, as you usually do when selecting text.
Then press the Copy button in the options bubble that appears to copy the selected text to the iOS clipboard.
You can now open any application in which it is possible to enter text, to paste the contents of the clipboard.
2. Recover text from Camera
The iOS 15 Camera can be used to quickly capture text displayed around you, without you having to take a photo.
To do this, open the Camera and point it at the panel, book, board, or any other medium that you come across and from which you want to extract the contents. When the camera detects the text of an object, it displays it in a very discreet yellow box.
Then tap on the text detection button at the bottom right of the screen. The Camera then displays the detected text in a pop-up in the center of the screen.
You can now finger select the content you need, then Copy it to iOS clipboard in order to paste it into another application.
3. Do a translation on the fly
If you are lucky enough to travel, but are not fluent in the local language, you can request a translation directly from the Camera. Detect and select text with the Camera app as seen in the previous step, then in the options menu, press the right arrow and choose to Translate it.
A Translate pane should then appear with the translation of the text you just submitted. Drag it up to fully deploy it and tap Change Language if you need to get a translation into another idiom.
4. Recover text from an image in Safari
In iOS 15, all images displayed in Safari can be used to extract text. To do this, hold your finger tap an image to preview it and tap the Show Text option.
The image then opens in a new page on a black background and automatically detects the text boxes.
You just have to select the elements you need to Copy, Find, Translate, etc.
5. Fill in text fields with the Camera
With iOS 15, you can use your iPhone’s Camera from any text field to capture an image and extract the text on the fly.
Take for example the creation of a new contact in iOS. Rather than manually entering the information displayed on a business card, you can use Live Text to do the work for you.
Tap and hold in a text field, then in the options bubble that appears, tap the Live text button.
The camera is then triggered. Point it at the text to retrieve, then when the text is correctly detected (yellow box), press Insert to fill the field with the retrieved information.
This function which allows you to retrieve text on the fly to insert it in a text zone is available in all applications. You can, for example, use it on Facebook or Twitter to post a message by directly scanning visible content around you.