We are obsessed with taking pictures. We capture moments of our lives so we can share them with others and store them for posterity. We take pictures of our kids, our pets, our vacations, and even the food we eat, creating a new phenomenon known as “the camera eats first.”
This obsession is reflected in the ever-increasing quality of smartphone cameras, as well as the popularity of photo-sharing and editing apps. Consider: 3.2 billion images are uploaded every day. When we run out of space to store all these pictures, services like iCloud save the day by offering additional cloud storage.
Unfortunately, this obsession comes at a cost. Every day, more and more sensitive and private information about us finds its way into our devices through our pictures, making us susceptible to hacking, theft, accidental disclosure, and ransom, among other risks. We are often unaware of what images are stored on our devices, whether they are accessible to generic apps, and if they are backed up automatically on cloud accounts with inadequate security protections.
These are not hypothetical concerns. In 2014, private pictures of many Hollywood celebrities were leaked online after their iCloud accounts were hacked. Recently, Jeff Bezos’s private pictures were leaked to a media company, allegedly by someone who had stolen them from their intended recipient, causing a major controversy. Most of us have perhaps experienced a time when we showed our vacation pictures to our colleagues, and they scroll one more picture to the left, revealing an embarrassing or private image.
In addition to private pictures, our phones also hold images of important documents like passports, tickets, tax forms, and medical prescriptions, presenting a new set of risks like identity theft and leakage of personal information. Surprisingly, we may not even remember taking some of these pictures in the first place.
It’s not only that this information can be accidentally disclosed or hacked by criminals. Many apps have access to a device’s entire photo library, and a bug in any such app has the potential to leak personal information.
This situation has only gotten worse during the pandemic, because most business is conducted online, and people are forced to click and share sensitive documents through their phones. We urgently need a solution that detects sensitive and private images on our phones and hides them away from curious onlookers, hackers, and buggy apps.
Norton Labs has developed a solution to this problem in the form of a new iOS app called SafePic. SafePic uses advanced machine learning to discover and help protect potentially sensitive images such as passports, social security cards, passwords, credit cards, and private photographs.
In its discovery phase, SafePic identifies these sensitive images, copies them to a separate secure vault, and depending on the user’s choice, either deletes the original images from the device’s photo library or replaces them with corresponding blurred placeholders. Original images can only be accessed through SafePic's vault or by selecting a placeholder in the device photo library and unblurring it with the help of SafePic’s extension.
Access to the vault or any unblurred image is permitted only if the user successfully authenticates with FaceID or their device passcode. When the user is not accessing the vault, images are encrypted with state-of-the-art cryptography to prevent backdoor access. A minimalist and elegant user interface allows users to interact with the app and find their images in the vault conveniently where they are grouped by the type of sensitive information they contain.
At the heart of SafePic is its detection engine, which quickly determines with high accuracy whether an image is sensitive. It also runs on a user’s device without relying on any cloud service. All detection is done locally on the device, and the user’s photos are not transferred off the phone unless explicitly requested using the iOS share menu. Custom-built by machine learning experts at our Norton Labs, this detection engine employs advanced classifiers based on convolutional neural networks and advanced optical character recognition (OCR). While this detection engine finds most common types of sensitive images, users also have an option to manually add pictures to the vault.
We realize that while certain pictures may require privacy from accidental disclosure, they do not merit the high-grade security of SafePic’s vault. For such images, we have developed a novel solution called PhotoBlur, which creates a blurred version of the photo when a user is swiping through the photo library. The user can then reveal the original, unblurred photo when they press and hold the screen. The PhotoBlur feature offers a nice balance between protection and ease-of-use for this class of picture.
How we created SafePic
SafePic’s detection engine consists of machine learning classifiers such as convolution neural networks for visual classification (VC) and optical character recognition (OCR), which are designed in-house by machine learning experts from the Norton Labs.
Training such classifiers to recognize sensitive images is challenging because, unlike other image classification tasks, there is no large existing database of such images. Publicly available images mainly consist of templates provided by agencies that issue sensitive documents, and they lack the diversity to simulate realistic conditions and the volume required to train data-hungry models like neural networks.
Our experts applied advanced techniques, including Transfer Learning and Data Augmentation, to build an accurate model with such limited data. In addition, because neural network-based visual classifiers work on very low-resolution images, they will flag any other image with the general visual characteristics of a sensitive document, causing false positives. Even at a false-positive rate of 5%, a user with 10,000 images on their phone would be left with 500 falsely identified images.
To address the issue of false positives, we leveraged OCR to identify the text in images and detect sensitive keywords that correspond to identity documents like “Passport,” “Date of Issue,” “Date of Birth,” etc. In contrast to a visual classifier, OCR requires higher definition images which causes a slowdown, not only due to increased processing, but also because phones often store only a low-resolution copy of the image on-device as a space saving measure. In order to apply OCR, the high-resolution image must be downloaded from the cloud.
Additionally, OCR is sensitive to the language of the document and does not easily generalize to documents in other languages, which the visual classifier does. Therefore, relying only on OCR leads to much slower scans and many false negatives.
To get the best of both worlds, we developed a pipeline that first detects a potential sensitive document, using the visual classifier and then perform specific OCR tests only on documents flagged as sensitive by the visual classifier. This ensures that we perform OCR tests on a much smaller set of images, preventing computation slowdown and network overhead, while ensuring high detection accuracy. Images that are flagged as sensitive by the visual classifier but where OCR didn’t find enough sensitive keywords are still presented to the user for review as potentially sensitive images. Overall, this hybrid approach achieves a very low false positive rate (approximately equal to 0.75%) and fast scan times (approximately equal to 180 images per minute).
We believe SafePic delivers on Norton’s vision to protect and empower people to live their digital lives safely and provides customers with an innovative solution to help keep their private photos private. Try SafePic out for yourself. It is available to download now in the Apple App Store.
As part of NortonLifeLock Inc., Norton Labs is leading the company’s future technology and guiding the consumer cybersecurity industry around the globe while delivering innovative prototypes with test-friendly features so adventurous users can learn and offer feedback."
Innovations from Norton Labs are for research, evaluation, and consumer feedback purposes. NortonLifeLock does not give any warranties as to the suitability or usability of these prototypes and recommends safeguarding data and reviewing all terms and conditions before use.
Copyright © 2021 NortonLifeLock Inc. All rights reserved. NortonLifeLock, the NortonLifeLock Logo, the Checkmark Logo, Norton, LifeLock, and the LockMan Logo are trademarks or registered trademarks of NortonLifeLock Inc. or its affiliates in the United States and other countries.
We encourage you to share your thoughts on your favorite social platform.