In this post, we will learn about how we can match two images by counting inlinears between them by technique of Large-Scale Image Retrieval with Attentive DEep Local Features (DELF).

So, Lets start

What is DEep Local Features (DELF) ? — The paper named ”Large-Scale Image Retrieval with Attentive Deep Local Features” proposed an attentive local feature descriptor suitable for large-scale image retrieval, referred to as DELF (DEep Local Feature). To evaluate the proposed descriptor, They introduce a new large-scale dataset, referred to as Google-Landmarks dataset, which involves challenges in both database and query such as background clutter, partial occlusion, multiple landmarks, objects in variable scales, etc.

DELF PIPELINE

four main blocks: dense localized feature extraction, keypoint selection, dimensionality reduction and indexing and retrieval.

Dataset used is Google-Landmarks dataset — It contains 1, 060, 709 images from 12, 894 landmarks, and 111, 036 additional query images. The images in the dataset are captured at various locations in the world, and each image is associated with a GPS coordinate.

In this article we will use a module that packages the DELF neural network and logic for processing images to identify keypoints and their descriptors.This is fully implemented and provided for users by tensorflow hub at here. The weights of the neural network were trained on images of landmarks as mentioned above.

What is a tensorflow hub? — TensorFlow Hub (TF-Hub) is a platform to share machine learning expertise packaged in reusable resources, notably pre-trained modules

Lets begin with the implementations,*(System prerequisites- python should be installed)*

(Source Code is in References section in the end)

1- Setting up the environment

As usual, we will create our virtual environment on python by typing the following command in the command line.(open command prompt in your code folder)

>virtualenv env (here “env” is the name of the environment, you can name anything here)

After creating the environment, go to the environment folder created in your code folder using cmd (in this case the environment folder is named env).

Go to /env/Scripts using cmd and hit the command

>activate.bat

Now, your environment is activated. Next step is to install some libraries

Use >pip install to install libraries in cmd. (Libraries you need to install are are — sklearn, pil, tensorflow)

2- Importing The Libraries

from absl import logging

import matplotlib.pyplot as plt

import numpy as np

from PIL import Image, ImageOps

from scipy.spatial import cKDTree

from skimage.feature import plot_matches

from skimage.measure import ransac

from skimage.transform import AffineTransform

from six import BytesIO

import tensorflow as tf

import tensorflow_hub as hub

from six.moves.urllib.request import urlopen

3- Taking URLs of images to be matched

IMAGE_1_URL\=’upload.wikimedia.org/wikipedia/commons/2/28..

IMAGE_2_URL\=’upload.wikimedia.org/wikipedia/commons/c/c3..

Images taken can be printed by,

plt.subplot(1,2,1)

plt.imshow(image1)

plt.subplot(1,2,2)

plt.imshow(image2)

OUTPUT-

4- Download the input images and resize it

def download_and_resize(name, url, new_width=256, new_height=256):

path = tf.keras.utils.get_file(url.split(‘/’)[-1], url)

image = Image.open(path)

image = ImageOps.fit(image, (new_width, new_height), Image.ANTIALIAS)

return image

image1 = download_and_resize(‘image_1.jpg’, IMAGE_1_URL)

image2 = download_and_resize(‘image_2.jpg’, IMAGE_2_URL)

5- Importing the model using tensorflow hub(link)

delf=hub.load(‘tfhub.dev/google/delf/1').signatures['defau..

The DELF module takes an image as input and will describe noteworthy points with vectors as you can see below. The following cell contains the core of this colab’s logic.

def run_delf(image):

np_image = np.array(image)

float_image = tf.image.convert_image_dtype(np_image, tf.float32)

return delf(image=float _image,score_threshold =tf.constant(100.0),image_scales=tf.constant([0.25, 0.3536, 0.5, 0.7071, 1.0, 1.4142, 2.0]),max_feature_num=tf.constant(1000))

result1 = run_delf(image1)

result2 = run_delf(image2)

Result1, result2 contains locations and description vectors

output from delf module

6- Matching of both the outputs is done and inliers are calculated

Now we will use the locations and description vectors to match the images

def match_images(image1, image2, result1, result2):

distance_threshold = 0.8

Read features.

num_features_1 = result1[‘locations’].shape[0]

print(“Loaded image 1’s %d features” % num_features_1)

num_features_2 = result2[‘locations’].shape[0]

print(“Loaded image 2’s %d features” % num_features_2)

Find nearest-neighbor matches using a KD tree.

d1_tree = cKDTree(result1[‘descriptors’])

_, indices = d1_tree.query(result2[‘descriptors’],

distance_upper_bound=distance_threshold)

Select feature locations for putative matches.

locations_2_to_use = np.array([
result2[‘locations’][i,]

for i in range(num_features_2)

if indices[i] != num_features_1

])

locations_1_to_use = np.array([

result1[‘locations’][indices[i],]

for i in range(num_features_2)

if indices[i] != num_features_1

])

# Perform geometric verification using RANSAC.

_, inliers = ransac(

(locations_1_to_use, locations_2_to_use),

AffineTransform,

min_samples=3,

residual_threshold=20,

max_trials=1000)

print(‘Found %d inliers’ % sum(inliers))

Visualize correspondences.

_, ax = plt.subplots()

inlier_idxs = np.nonzero(inliers)[0]

plot_matches(

ax,

image1,

image2,

locations_1_to_use,

locations_2_to_use,

np.column_stack((inlier_idxs, inlier_idxs)),

matches_color=’b’)

ax.axis(‘off’)

ax.set_title(‘DELF correspondences’)

match_images(image1, image2, result1, result2)

Result of matching function looks like this,

Found 51 inliers

Summary

We learned what is DEep Local Features, how we can use its implementation by tensorflow hub into our own implementation in python code for matching our images.

References

• Large-Scale Image Retrieval with Attentive Deep Local Features, Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, Bohyung Han
• SOURCE CODE-https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/tf_hub_delf_module.ipynb#scrollTo=f3nk38tIKytQ

Congrats! You’ve made it

do write comments

Have fun, keep learning, and always keep coding

My LinkedIn & Twitter