Amazon Cognito Triggers: Sign In Users With Their Face | Part 2

We will start from where we left last time, so, if you didn't read the first part and you want to catch up, Amazon Cognito Triggers: All You Need To Know About Them | Part 1. I'll write down only the parts relating to this article so I won't explain all triggers but only the ones missing from the first part.

Introduction

We will walk through the process of signing up and signing in users capturing their face image, all this is done to show you how the following triggers work:

• Define Auth challenge trigger: we use this trigger to define our custom challenge. We can give a name to our challenge, find how many attempts Frodo has done and fail his authentication process and, if Frodo answers correctly to our challenge, we can notify Cognito to issue the token;
• Create Auth challenge trigger: it creates our custom challenge, aka: it finds the right answer and stores it for later, moreover it could give users some parameters to use to answer the challenge;
• Verify Auth challenge response trigger: it verifies answers from users and notify Cognito if those answers are right✅.

Solution overview

Users sign up using a custom web application which uses Amplify to ease Cognito integration. After they sign up using only their email address, they need to confirm their address by submitting the verification code received, then the web application asks them to capture a photo from the webcam and send it to complete the sign up. When the picture is stored in S3, from the signup process, it triggers a lambda function which indexes the face into Amazon Rekognition and save the face id into the Amazon DynamoDB users table.

Users sign in typing their email, the web application asks them to capture a photo of themselves; if their face matches the one they previously submitted, they are successfully logged in 🎉.

Now that we know the behavior we can start to deep dive into the solution. There are two main components:

• Frontend: a React web application to help users with the sign up and sign in process;
• Backend: fully serverless, as seen in the part 1, it leverages Cognito lambda triggers, S3 and S3 lambda triggers; moreover it make use of Rekognition to index and search users by their face and DynamoDB to store users data.

Amazon Rekognition makes it easy to add image and video analysis to your applications. You just provide an image or video to the Amazon Rekognition API, and the service can identify objects, people, text, scenes, and activities. It can detect any inappropriate content as well. Amazon Rekognition also provides highly accurate facial analysis, face comparison, and face search capabilities. You can detect, analyze, and compare faces for a wide variety of use cases, including user verification, cataloging, people counting, and public safety.

Alright let's explore the backend and then the frontend, I won't go deeper into the frontend because my knowledge is basic hence I don't want to embarrass myself 😂.

Backend

Requirements

Well if you read the first part you don't need to do anything more, meanwhile if you didn't read…what are you waiting for, click the link below.

Amazon Cognito Triggers: All You Need To Know About Them | Part 1

Perfect by now you know how to create and link triggers to Cognito using CDK, so we will skip that and go straight to Amazon Rekognition and the resources needed in our infrastructure. Let's create our Amazon Rekognition collection

  this.faceCollection = new CfnCollection(this, name, {
    collectionId: name,
  });

then we need to create the lambda which will be triggered by S3, as you can see we import the bucket from its arn and then we add a trigger when object are created within the private/ folder

  const indexFaces = this.createLambdaFunction(name(`${id}-face-indexing`), 'face-indexing', baseEnv);

  this.usersBucket = Bucket.fromBucketArn(this, name(`${id}`), props.usersBucketArn);
  this.usersBucket.addEventNotification(EventType.OBJECT_CREATED, new LambdaDestination(indexFaces), { prefix: 'private/' });

As per the lambda itself what we need to do is get the S3 key and use it to index the face detected in the image, to do so we use the rekognition class from aws

  //key format: /private/{region}:{identity_pool_id}/{user_sub}.jpeg
  const key = decodeURIComponent(record.s3.object.key);
  const bucketName = record.s3.bucket.name;
  const sub = key.split('/')[2].replace(`.jpeg`, '');
  const params = {
    CollectionId: process.env.COLLECTION_ID,
    ExternalImageId: sub,
    Image: { S3Object: { Bucket: bucketName, Name: key } },
    MaxFaces: 1,
  }      
  const indexRes = await rekognition.indexFaces(params).promise();
  console.info(`Successfully indexed face for user ${sub}`);
  console.info(`Indexing confidence is ${indexRes.FaceRecords[0].Face.Confidence}`);

Disclaimer: there will be an issue when users do not send a valid image, or it won't have enough confidence to be saved, because the process is asynchronous and we cannot give any type of feedback to the users. You can work around the issue by: 👉 using an API instead of S3 trigger and Amazon Identity Pool 👉 asking users to submit a valid sign up image when they try to sign in and the trigger won't find their image indexed (maybe we could also ask to verify their identity by sending a code to their inbox). Why i did not do it? Because I really wanted to try Amazon Identity Pool from the web application🙏.

And as simple as that we have our infrastructure almost ready, now let's review the sign up and sign in process.

Sign up

As we can see it follows the standard sign up flow, but in addition we do have a lambda trigger to S3 which indexes the face thanks to Rekognition and then save the face id to our database. This is pretty standard, let's move on to the core of this solution 🤟.

Sign in

Let's review the flow:

1. users try to sign in, which triggers the "Define Auth Challenge": • defines our custom challenge; • fails the authentication after 3 wrong attempts; • issues the token when the answer is correct.

2. then Cognito calls the "Create Auth Challengef: • fetches the face id from our users table; • gets the Presigned url from S3, which returns an URL the application can use to upload the image without users being authenticated. The cool thing is we can specify the exact path and file name we want.

  const result = await dynamodb.get(paramsGet).promise

  const key = `signin/${event.request.userAttributes.sub}/${crypto.randomUUID()}.jpeg`;
  const presignedUrl = await s3.getSignedUrlPromise('putObject', {
    Bucket: process.env.USERS_BUCKET,
    Key: key,
    Expires: 60 * 10,
  });

  event.response.publicChallengeParameters = { presignedUrl, key, };
  event.response.privateChallengeParameters = { answer: result.Item.faceId };

• publicChallengeParameters: users receive those parameters to complete the custom challenge, in this case we gave to them a presigned url to upload their image and a key to answer the challenge;
• privateChallengeParameters: those parameters are stored within Cognito and passed through to the verify challenge trigger. Users do not have access to them.

3. users, having received our presigned url, can now upload their picture and send the response to our custom challenge. Cognito then calls the "Verify Auth Challenge Response" which search a match using the face detected in the picture.

  const params = {
    CollectionId: process.env.COLLECTION_ID,
    Image: {
      S3Object: {
        Bucket: process.env.USERS_BUCKET,
        Name: event.request.challengeAnswer,
      }
    },
    MaxFaces: 1,
    FaceMatchThreshold: 95
  };
  const res = await rekognition.searchFacesByImage(params).promise();

  for(const match of res.FaceMatches) {
    if(event.request.privateChallengeParameters.answer === match.Face.FaceId) {
      event.response.answerCorrect = true;
    }
  }

We have also specified a value for the confidence of the findings, doing so it automatically discard results which do not meet our confidence level. If there is a match then the answer is correct and our users will successfully sign in 🥳.

Frontend

First of all I want to thank my dear friend Gianmarco who helped me porting the application from JS to TS and refactoring my first """working""" version, he's probably still laughing at my code🙈.

The interesting part about the web application is Amazon Amplify integrated with Amazon Cognito for signing in, signing up and also for uploading files to S3. There are two types of uploads to S3:

• Authenticated: after successfully signing up, the application ask users to capture their image, which is sent leveraging Cognito Identity Pools, which means we create a IAM role for authenticated users and they will use it to upload directly to S3 without any API;
• Unauthenticated: when users are trying to log in, they need to send their image, as they are unauthenticated we use a handy feature called S3 Presigned Url which allows us to create a link which can be used to make a PUT request and save the image users captured. Yeah it's that simple😉.

Requirements

I've started the project googling "getting started with reactjs", so keep it in mind when you see my code snippets 🙃.

First we need to install some standard react libraries, after doing that we need to install Amplify as local library and also as global package because we will use its cli:

npm install -g @aws-amplify/cli

Let's configure amplify and import our S3 bucket and Cognito identity and users pool with the followig lines:

  amplify init
  amplify import auth
  amplify import storage
  amplify push

If we imported our resources successfully we should see a new files called aws-exports.js which contains all our resources reference. Now we need to use it in our app:

  import { Amplify } from 'aws-amplify';
  import awsmobile from './aws-exports';

  Amplify.configure(awsmobile)

Perfect we are ready to go and the react application should compile successfully.

Sign up

We collect users email with this simple form, when they press the button we interact with amplify:

  await Auth.signUp({
    username,
    password: `zZ8${Math.random().toString(36).slice(-8)}!`,
    attributes: { email: username, },
    autoSignIn: { enabled: true, },
  });

As you can see we did two things:

• we have to specify a password to create users, I did it really fast as you can see it (don't judge 👀);
• we enable auto sign in so, after we confirm our email, we are automatically signed in without asking one more time username and password to our users (not that we could have, they don't know their password).

Users receive in their inbox a verification code, after they confirm their email we call amplify to confirm them

await Auth.confirmSignUp(username, verifyCode);

Perfect, our users can now confirm themselves. Onto the next phase: the application ask users to capture and then submit a picture of themselves using the integrated webcam.

This page asks users to capture their photo with the webcam.

Yeah I'm rocking some Cloud Guru swag right, not sponsored though 😞.

When users submit their own image we call the Amplify Storage class which helps us uploading the image into S3

  await Storage.put(id + '.jpeg', cameraFile, {
    level: 'private',
    contentType: 'image/jpeg',
  });

Two things I want to point out: with this 4 lines we are using Amazon Identity Pools which means Amplify handles the communication with Cognito which retrieves our temporary credentials and use those to upload our file to S3. The second one regards the level object:

• private: it uploads the image to S3 into a folder named /private/{{region}}:{{identity_pool_id}}/{{file_name}};
• public: it saves the image to S3 into a folder named /public/{{file_name}}.

Sign in

For end users this process is the same as the sign up, but actually the logic beneath the application is a bit different.

First we try to sign in users using only their email, the result let us know we need to execute the custom challenge flow:

  const cognitoUser = await Auth.signIn(username);

  if (cognitoUser.challengeName === 'CUSTOM_CHALLENGE') {
    setState("CustomChallenge");
    setchallengeRequest(cognitoUser);
  }

the result is stored within a variable challengeRequest and contains the backend-generated information such as:

• S3 presigned url to upload file to S3;
• S3 file key to respond to the challenge. Using this data we can answer Cognito's custom challenge, if the face match then users go to the success page, else the application ask them to retake the photo

  await fetch(challengeRequest.challengeParam.presignedUrl, {
    method: "PUT",
    body: cameraFile,
  });

  const cognitoUser = await Auth.sendCustomChallengeAnswer(challengeRequest, challengeRequest.challengeParam.key);

  if (cognitoUser.authenticationFlowType !== 'CUSTOM_AUTH') {
    setState("Success");
  } else {
    setState("CustomChallenge");
    setchallengeRequest(cognitoUser);
    setCameraUrl(null);
    setCameraFile(undefined);
  }

And this wraps up the frontend part.

Conclusion

Well first of all I have to thank again Gianmarco for making the frontend easier to read and understand for everybody👏.

This blogpost was really fun to make, I had the chance to show you how the Cognito custom auth challenge triggers work and I did it with an interesting example using Amazon Rekognition. As per the solution itself I must say, as cool as it seems, it probably does not achieve regulatory compliance, mainly because you can hold a picture of someone and it still count as a face so if someone knows your email could try to access your account with one of your images. Still it's a fun alternative to sign up.

You can find the project here https://github.com/Depaa/cognito-triggers 😉 Keep in mind there is also the part one of my Cognito series.

Thank you so much for reading! 🙏 I will keep posting different AWS architecture from time to time so follow me on LinkedIn 👉 https://www.linkedin.com/in/matteo-depascale/.