Finding the nearest locations around you using AWS Amplify — Part 2

Introduction to Amazon Elasticsearch service

At the end of Part 1 of this series, you changed your GraphQL Schema to include location coordinates and added @searchable to a type to enable distance-aware searches.

Introduction to open source Elasticsearch

Elasticsearch is an open source search and analytics engine based on the Lucene library. By using Elasticsearch we can provide advanced search capabilities to our users including the following:

  • Natural language searches: using fuzzy matching, multi match, custom boosting, logical operators, wildcards, regular expressions and range queries.
  • Advanced searches: using terms (structured searches), filters, suggestions, custom relevance score and faceted (Eg: via categories).
  • Geolocation searches: distance from a location or within a custom region (bounding box or polygon).
Searches by distance or within a custom region (using a bounding box or polygon).

Elasticsearch architecture

Find an overview of the Elasticsearch architecture in the diagram below. We will introduce Kibana later on.

Elasticsearch Architecture Overview (source elastic.co)

Elasticsearch cluster and data nodes (servers)

An Elasticsearch cluster (domain) is composed by one or more data nodes working together. A data node is a running instance and usually hosted in one server.

Indexes and mappings (databases and schemas)

An Index works like a database with mappings that hold schema definitions for its internal types. An Index is a logical namespace mapped to one or more shards (primary and replicas).

type BikePoint @model @searchable {
id: ID!
name: String!
description: String
location: Location
bikes: Int
}

Types, fields and documents (tables, columns and rows)

A type works like a table. Each type, has a list of fields and mappings that tell Elasticsearch how to analyse and store them properly. A field works like a column and can contain scalars or complex structures. A document works like a row, each document is stored as a JSON object in the _sourcefield and returned in searches.

  • Index. Processing a document and storing it in an index for retrieval.
  • Delete. Removing a document from an index.
  • Update. Removing a document and indexing it as a new document.
  • Search. Retrieving documents or aggregates from one or more indices.

Finding your new Amazon Elasticsearch domain

In order to access your Amazon Elasticsearch domain, you can run the following command using Amplify CLI selecting GraphQL.

amplify console api
Data Sources for LondonCycles app.
Amazon Elasticsearch Dashboard with a new domain created by @searchable.
Amazon Elasticsearch domain details.

Kibana Developer Tools

Along with Elasticsearch, you have access to Kibana Developer Tools. This is an environment you can use to test your search queries. Let’s see how you can access it.

$ cat ~/.aws/credentials                           (Linux & Mac)
$ %USERPROFILE%\.aws\credentials > con (Windows)
[default]
aws_access_key_id=<<KEY-ID>>
aws_secret_access_key=<<SECRET-ACCESS-KEY>>

Step 1: setting up a distance-aware Elasticsearch index

To enable distance-aware searches, we need to make sure Elasticsearch is properly setup. Before we make further changes, we need to:

  • Create a new index: use the name of your type, all letters must be in lowercase. Eg: bikepoint.
  • Create a new geolocation mapping: this is so latitude and longitude fields are indexed properly as a geo_point.
# Create a new index
PUT /bikepoint
# Create a new mapping to set location type to geo_point
PUT /bikepoint/_mapping/doc
{
"properties": {
"location": {
"type": "geo_point"
}
}
}

Step 2: creating a custom GraphQL distance-aware search query

In Part 1, we have seen how to use the default GraphQL search query that AWS AppSync creates. In this section, we will create a custom GraphQL distance-aware search to find items nearby within a distance from a given location.

Find bikes nearby feature in LondonCycles app.
type Query {
nearbyBikeStations(
location: LocationInput!,
m: Int,
limit: Int,
nextToken: String
): ModelBikePointConnection
}
input LocationInput {
lat: Float!
lon: Float!
}
type ModelBikePointConnection {
items: [BikePoint]
total: Int
nextToken: String
}
  • location: origin coordinates from where we want to run our search.
  • m: maximum distance from location in meters. Default: 500 meters.
  • limit: how many results we want back. Default: 10 results.
  • nextToken: distance from which to paginate from, used for pagination.

Step 3: creating a custom AWS AppSync resolver

We will now create a custom AWS AppSync resolver for Amazon Elasticsearch service. This will allow you to use GraphQL to store and retrieve data from the Amazon Elasticsearch domain created by the @searchable GraphQL transform in Part 1.

Custom GraphQL distance-aware search query flow
/amplify/backend/api/
TransportForLondonAPI/resolvers
Query.nearbyBikeStations.req.vtl
Query.nearbyBikeStations.res.vtl
query NearbyBikes {
nearbyBikeStations(
location: { lat: 51.510239, lon: -0.134167 },
m: 500,
limit: 3
) {
items {
id
name
location { lat lon }
}
total
}
}
## Query.nearbyBikeStations.req.vtl#set( $distance = $util.defaultIfNull($ctx.args.m, 500) )
#set( $limit = $util.defaultIfNull($ctx.args.limit, 10) )
{
"version": "2017-02-28",
"operation": "GET",
"path": "/bikepoint/doc/_search",
"params": {
"body": {
#if( $context.args.nextToken )"search_after": ["$context.args.nextToken"], #end
"size" : ${limit},
"query": {
"bool" : { "must" : { "match_all" : {} },
"filter" : {
"geo_distance" : {
"distance" : "${distance}m",
"distance_type": "arc",
"location" : $util.toJson($ctx.args.location) }
}

}
},
"sort": [{
"_geo_distance": {
"location": $util.toJson($ctx.args.location),
"order": "asc",
"unit": "m",
"distance_type": "arc"
}

}]
}
}}
#set( $distance = $util.defaultIfNull($ctx.args.m, 500) )
#set( $limit = $util.defaultIfNull($ctx.args.limit, 10) )
"operation": "GET",  
"path": "/bikepoint/doc/_search",
  • search-after last sort value for pagination purposes.
  • size, number of results.
  • query, query details and filters.
  • sort, sorting fields.
"query": {        
"bool" : {
"must" : { "match_all" : {} },
"filter" : {
"geo_distance" : {
"distance" : "${distance}m",
"distance_type": "arc",
"location" : $util.toJson($ctx.args.location)
}
}

}
}
"sort": [{ 
"_geo_distance": {
"location": $util.toJson($ctx.args.location),
"order": "asc",
"unit": "m",
"distance_type": "arc"
}
}]
Example search query and results.
## Query.nearbyBikeStations.res.vtl#set( $items = [] )
#foreach( $entry in $context.result.hits.hits )
#if( !$foreach.hasNext )
#set( $nextToken = "$entry.sort.get(0)" )
#end
$util.qr($items.add($entry.get("_source")))
#end
$util.toJson({
"items": $items,
"total": $ctx.result.hits.total,
"nextToken": $nextToken
})
{
"took": 17,
"hits": {
"total": 12,
"hits": [
{
"_index": "bikepoint",
"_type": "doc",
"_id": "BikePoints_83",
"_source": {
"id": "BikePoints_83",
"__typename": "BikePoint",
"name": "Panton Street, West End",
"location": {
"lon": -0.13151,
"lat": 51.509639
},
"createdAt": "2020-03-02T14:48:44.617Z",
"updatedAt": "2020-03-02T14:48:44.617Z"
},
"sort": [
195.6063243635008
]
}
]
}
}

Paginating results from Elasticsearch using AWS AppSync

Let’s see an example using the pagination implemented in our custom AWS AppSync resolver for thenearbyBikeStations query.

////////////////////////////////////////////////////////////////////
// First query. We grab nextToken from results
nearbyBikeStations(
location: { lat: 51.510239, lon: -0.134167 }, m: 500, limit: 1
) {
items { id name location { lat lon } }
total
nextToken
}
////////////////////////////////////////////////////////////////////
// Query sent to Elasticsearch
GET /bikepoint/doc/_search
{
"size" : 1,
"query": {...},
"sort": [...]
}
////////////////////////////////////////////////////////////////////
// Result
{
"took": 25,
"hits": {
"total": 12,
"hits": [
{
"_index": "bikepoint",
"_type": "doc",
"_id": "BikePoints_83",
"_source": {
"__typename": "BikePoint",
"name": "Panton Street, West End",
"location": {
"lon": -0.13151,
"lat": 51.509639
},
"id": "BikePoints_83",
},
"sort": [
195.6063243635008
]
}
]
}
}
////////////////////////////////////////////////////////////////////
// Second query as user paginates. We use previous nextToken
nearbyBikeStations(
location: { lat: 51.510239, lon: -0.134167 }, m: 500, limit: 1,
nextToken: "195.6063243635008"
) {
items { id name location { lat lon } }
total
nextToken
}
////////////////////////////////////////////////////////////////////
// Query sent to Elasticsearch
GET /bikepoint/doc/_search
{
"search_after": [195.6063243635008],
"size" : 1,
"query": {...},
"sort": [...]
}

Step 4: adding custom AWS AppSync resolver to your Amplify project

Using Amplify we can add a custom AWS AppSync resolver by adding its templates to our project. These will be part of CustomResources.jsonin your apistack.

/amplify/backend/api/
TransportForLondonAPI/stacks
CustomResources.json
{
"Resources": {
"QueryNearbyBikeStations": {
"Type": "AWS::AppSync::Resolver",
"Properties": {
"ApiId": { "Ref": "AppSyncApiId" },
"DataSourceName": "ElasticSearchDomain",
"TypeName": "Query",
"FieldName": "nearbyBikeStations",
"RequestMappingTemplateS3Location": {
"Fn::Sub": [
"s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.nearbyBikeStations.req.vtl", {
"S3DeploymentBucket": { "Ref": "S3DeploymentBucket" },
"S3DeploymentRootKey": { "Ref": "S3DeploymentRootKey" }
}]
},
"ResponseMappingTemplateS3Location": {
"Fn::Sub": [ "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/Query.nearbyBikeStations.res.vtl", {
"S3DeploymentBucket": { "Ref": "S3DeploymentBucket" },
"S3DeploymentRootKey": { "Ref": "S3DeploymentRootKey" }
}]
}
}
},
}

Pushing changes to the cloud

Run the following command to push all changes to the cloud:

amplify push

Conclusion

Congratulations! You have learnt how to create a custom GraphQL distance-aware search using AWS Amplify and AWS AppSync together with Amazon Elasticsearch.

Ready to code?

You don’t have an AWS Account? Use the next few minutes to create one and activate the free plan for a whole year. Follow steps at AWS knowledge center.

Free tier for a new AWS Account. Check out latest pricing.
Picture with Jane Shih from last Open Up Summit in Taipei, Taiwan.

Thanks for reading!

Have you got any questions regarding this article AWS Amplify or AWS AppSync? Feel free to ping me anytime at @gerardsans.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Gerard Sans

Gerard Sans

Helping Devs to succeed #AI #web3 / ex @AWSCloud / Just be AWSome / MC Speaker Trainer Community Leader @web3_london / @ReactEurope @ReactiveConf @ngcruise