Rotates Amazon Personalize filters on a schedule based on dynamic templates

Overview

Amazon Personalize Filter Rotation

This project contains the source code and supporting files for deploying a serverless application that provides automatic filter rotation capabilities for Amazon Personalize, an AI service from AWS that allows you to create custom ML recommenders based on your data. Highlights include:

  • Creates filters based on a dynamic filter naming template you provide
  • Builds filter expressions based on a dynamic filter expression template you provide
  • Deletes filters based on a dynamic matching expression you provide (optional)
  • Publishes events to Amazon EventBridge when filters are created or deleted (optional)

Why is this important?

Amazon Personalize filters are a great way to have your business rules applied to recommendations before they are returned to your application. They can be used to include or exclude items from being recommended for a user based on a SQL-like syntax that considers the user's interaction history, item metadata, and user metadata. For example, only recommend movies that the user has watched or favorited in the past to populate a "Watch again" widget.

INCLUDE ItemID WHERE Interactions.event_type IN ('watched','favorited')

Or exclude products from being recommended that are currently out of stock.

EXCLUDE ItemID WHERE Items.out_of_stock IN ('yes')

You can even use dynamic filters where the filter expression values are specified at runtime. For example, only recommend movies for a specific genre.

INCLUDE ItemID WHERE Items.genre IN ($GENRES)

To use the filter above, you would pass the appropriate value(s) for the $GENRE variable when retrieving recommendations.

You can learn more about filters on the AWS Personalize blog here and here.

Filters are great! However, they do have some limitations. One of those limitations is being able to specify a dynamic value for a range query (i.e., <, <=, >, >=). For example, the following filter to limit recommendations to new items that were created since a rolling point in the past is not supported.

THIS WON'T WORK!

INCLUDE ItemID WHERE Items.creation_timestamp > $NEW_ITEM_THRESHOLD

The solution to this limitation is to use a filter expression with a hard-coded value for range queries.

THIS WORKS!

INCLUDE ItemID WHERE Items.creation_timestamp > 1633240824

However, this is not very flexible or maintainble since time marches on but your static filter expression does not. The workaround is to update your filter expression periodically to maintain a rolling window of time. Unfortunately filters cannot be updated so a new filter has to be created, your application has to transition to using the new filter, and then the previous filter can be safely deleted.

The purpose of this serverless application is to make this process easier to maintain by automating the creation and deletion of filters and allowing you to provide a dynamic expression that is resolved to the appropriate hard-coded value when the new filter is created.

Here's how it works

An AWS Lambda function is deployed by this application that is called on a recurring basis. You control the schedule which can be a cron expression or a rate expression. The function will only create a new filter if a filter does not already exist that matches the current filter name template and it will only delete existing filters that match the delete template. Therefore, it is fine to have the function run more often than necessary (i.e. if you don't have a predictable and consistent time when filters should be rotated).

The key to the filter rotation function are the templates used to verify that the current template exists and if existing template(s) are eligible to delete. Let's look at some examples. You provide these template values as CloudFormation parameters when you deploy this application.

Current filter name template

Let's say you want to use a filter that only recommends recently created items. The CREATION_TIMESTAMP column in the items dataset is a convenient field to use for this. This column name is reserved and is used to support the cold item exploration feature of the aws-user-personalization recipe. Values must be expressed in the Unix timestamp format as long's (i.e. number of seconds since the Epoch). The following expression limits items that were created in the last month (1633240824 is the Unix timestamp from 1 month ago as of this writing).

INCLUDE ItemID WHERE Items.creation_timestamp > 1633240824

Alternatively, you can use a custom metadata column for the filter that uses a more coarse and/or human readable format but is still comparable for range queries, like YYYYMMDD.

INCLUDE ItemID WHERE Items.published_date > 20211001

As noted earlier, filters cannot be updated. Therefore you can't just change the filter expression of the filter. Instead, you have to create a new filter with a new expression, switch your application to use the new filter, and then delete the old filter. This requires using a predictable naming standard for filters so applications can automatically switch to using the new filter without a coding change. Continuing with the creation timestamp theme, the filter name could be something like.

filter-include-recent-items-20211101

Assuming we want to rotate this filter each day, the next day's filter name would be filter-include-recent-items-20211004, then filter-include-recent-items-20211005, and so on as time passes. Since there are limits on how many active filters you can have at any time, you cannot precreate filters. Instead, this application will dynamically create new filters. What is needed is a template that defines the filter name that can be resolved when the rotation script runs.

filter-include-recent-items-{{datetime_format(now,'%Y%m%d')}}

The above filter name template will resolve and replace the expression within the {{ and }} characters (handlebars or mustaches). In this case, we are taking the current time expressed as now and formatting it using the %Y%m%d date format expression. The result (as of today) is 20211102. If the rotation function finds an existing filter with this name, a new filter does not need to be created. Otherwise, a new filter is created using filter-include-recent-items-20211102 as the name.

The PersonalizeCurrentFilterNameTemplate CloudFormation template parameter is how you specify your own custom filter name template.

The functions and operators available to use in templates is described below.

Current filter expression template

When rotating and creating the new filter, we also may have to dynamically resolve the actual filter expression. The PersonalizeCurrentFilterExpressionTemplate CloudFormation template parameter can be used for this. Some examples.

INCLUDE ItemID WHERE Items.CREATION_TIMESTAMP > {{int(unixtime(now - timedelta_days(30)))}}
INCLUDE ItemID WHERE Items.published_date > {{datetime_format(now - timedelta_days(30),'%Y%m%d')}}

The above templates produce a hard-coded filter expression based on current time when they're resolved. The first produces a Unix timestamp (expressed in seconds as required by Personalize for CREATION_TIMESTAMP) that is 30 days ago. The second template produces an integer representing the date in YYYYMMDD format from 30 days ago.

Delete filter match template

Finally, we need to clean up old filters after we have transitioned to a newer version of the filter. A filter name matching template can be used for this and can be written in such a way to delay the delete for some time after the new filter is created. This gives your application time to transition from the old filter to the new filter. The PersonalizeDeleteFilterMatchTemplate CloudFormation template parameter is where you specify the delete filter match template.

The following delete filter match template will match on filters with a filter name that starts with filter-include-recent-items- and has a suffix that is more than one day older than today. In other words, we have 1 day to transition to the new filter before the old filter is deleted.

starts_with(filter.name,'filter-include-recent-items-') and int(end(filter.name,8)) < int(datetime_format(now - timedelta_days(1),'%Y%m%d'))

Any filters that trigger this template to resolve to true will be deleted. All others will be left alone. Note that all fields available in the FilterSummary of the ListFilters API response are available to this template. For example, the template above matches on filter.name. Other filter summary fields such as filter.status, filter.creationDateTime, and filter.lastUpdatedDateTime can also be inspected in the template.

Filter events

If you'd like to synchronize your application's configuration or be notified when a filter is created or deleted, you can optionally configure the rotator function to publish events to Amazon EventBridge. When events are enabled, there are two event detail types published by the rotator function: PersonalizeFilterCreated and PersonalizeFilterDeleted. Both have an event Source of personalize.filter.rotator and include details on the filter created or deleted. This allows you to setup EventBridge rules to process events as you please. For example, when a new filter is created, you can receive the PersonalizeFilterCreated event in a Lambda function to update your application's configuration to switch to using the new filter.

Filter template syntax

The Simple Eval library is used as the foundation of for the template syntax. It provides a safer and more sandboxed alternative than using Python's eval function. Check the Simple Eval library documentation for details on the functions available and examples.

The following additional functions were added as part of this application to make writing templates easier for rotating filters.

  • unixtime(value): Returns the Unix timestamp value given a string, datetime, date, or time. If a string is provided, it will be parsed into a datetime first.
  • datetime_format(date, pattern): Formats a datetime, date, or time using the specified pattern.
  • timedelta_days(int): Returns a timedelta for a number of days. Can be used for date math.
  • timedelta_hours(int): Returns a timedelta for a number of hours. Can be used for date math.
  • timedelta_minutes(int): Returns a timedelta for a number of minutes. Can be used for date math.
  • timedelta_seconds(int): Returns a timedelta for a number of seconds. Can be used for date math.
  • starts_with(str, prefix): Returns True if the string value starts with prefix.
  • ends_with(str, suffix): Returns True if the string value ends with suffix.
  • start(str, num): Returns the first num characters of the string value
  • end(str, num): Returns the last num characters of the string value
  • now: Current datetime

Installing the application

IMPORTANT NOTE: Deploying this application in your AWS account will create and consume AWS resources, which will cost money. The Lambda function is called according to the schedule you provide but typically should not need to be called more often than hourly. Personalize does not charge for filters but your account does have a limit on the number of filters that are active at any time. There are also limits on how many filters can be in a pending or in-progress status at any point in time. Therefore, if after installing this application you choose not to use it as part of your solution, be sure to follow the Uninstall instructions in the next section to avoid ongoing charges and to clean up all data.

This application uses the AWS Serverless Application Model (SAM) to build and deploy resources into your AWS account.

To use the SAM CLI, you need the following tools installed locally.

To build and deploy the application for the first time, run the following in your shell:

sam build --use-container --cached
sam deploy --guided --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

If you receive an error from the first command about not being able to download the Docker image from public.ecr.aws, you may need to login. Run the following command and then retry the above two commands.

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

The first command will build the source of the application. The second command will package and deploy the application to your AWS account, with a series of prompts:

Prompt/Parameter Description Default
Stack Name The name of the stack to deploy to CloudFormation. This should be unique to your account and region. personalize-filter-rotator
AWS Region The AWS region you want to deploy this application to. Your current region
Parameter PersonalizeDatasetGroupArn Amazon Personalize dataset group ARN to rotate filters within.
Parameter PersonalizeCurrentFilterNameTemplate Template to use when checking and creating the current filter.
Parameter PersonalizeCurrentFilterExpressionTemplate Template to use when building the filter expression when creating the current filter.
Parameter PersonalizeDeleteFilterMatchTemplate (optional) Template to use to match existing filters that should be deleted.
Parameter RotationSchedule Cron or rate expression to control how often the rotation function is called. rate(1 day)
Parameter Timezone Set the timezone of the rotator function's Lambda environment to match your own.
Parameter PublishFilterEvents Whether to publish events to the default EventBridge bus when filters are created and deleted. Yes
Confirm changes before deploy If set to yes, any CloudFormation change sets will be shown to you before execution for manual review. If set to no, the AWS SAM CLI will automatically deploy application changes.
Allow SAM CLI IAM role creation Since this application creates IAM roles to allow the Lambda functions to access AWS services, this setting must be Yes.
Save arguments to samconfig.toml If set to yes, your choices will be saved to a configuration file inside the application, so that in the future you can just re-run sam deploy without parameters to deploy changes to your application.

TIP: The SAM command-line tool provides the option to save your parameter values in a local file (samconfig.toml) so they're available as defaults the next time you deploy the app. However, SAM wraps your parameter values in double-quotes. Therefore, if your template parameter values contain embedded string values (like the date format expressions shown in the examples above), be sure to use single-quotes for those embedded values. Otherwise, your parameter values will not be properly preserved.

Uninstalling the application

To remove the resources created by this application in your AWS account, use the AWS CLI. Assuming you used the default application name for the stack name (personalize-filter-rotator), you can run the following:

aws cloudformation delete-stack --stack-name personalize-filter-rotator

Alternatively, you can delete the stack in CloudFormation in the AWS console.

Reporting issues

If you encounter a bug, please create a new issue with as much detail as possible and steps for reproducing the bug. Similarly, if you have an idea for an improvement, please add an issue as well. Pull requests are also welcome! See the Contributing Guidelines for more details.

License summary

This sample code is made available under a modified MIT license. See the LICENSE file.

You might also like...
Collection of AWS Fault Injection Simulator (FIS) experiment templates.

Collection of AWS Fault Injection Simulator (FIS) experiment templates. These templates let you perform chaos engineering experiments on resources (applications, network, and infrastructure) in the AWS Cloud.

Cookiecutter templates for Serverless applications using AWS SAM and the Rust programming language.

Cookiecutter SAM template for Lambda functions in Rust This is a Cookiecutter template to create a serverless application based on the Serverless Appl

A simple Python wrapper for the Amazon.com Product Advertising API ⛺

Amazon Simple Product API A simple Python wrapper for the Amazon.com Product Advertising API. Features An object oriented interface to Amazon products

The unofficial Amazon search CLI & Python API
The unofficial Amazon search CLI & Python API

amzSear The unofficial Amazon Product CLI & API. Easily search the amazon product directory from the command line without the need for an Amazon API k

A simple library for interacting with Amazon S3.

BucketStore is a very simple Amazon S3 client, written in Python. It aims to be much more straight-forward to use than boto3, and specializes only in

The algorithm performs a simple user registration (Name, CPF, E-mail and Telephone) in an Amazon RDS database and also performs the storage, training and facial recognition of the user's face to identify the users already registered in the system in a next time the user is seen.
Integrating Amazon API Gateway private endpoints with on-premises networks

Integrating Amazon API Gateway private endpoints with on-premises networks Read the blog about this application: Integrating Amazon API Gateway privat

HTTP Calls to Amazon Web Services Rest API for IoT Core Shadow Actions 💻🌐💡

aws-iot-shadow-rest-api HTTP Calls to Amazon Web Services Rest API for IoT Core Shadow Actions 💻 🌐 💡 This simple script implements the following aw

 Automated endpoint management for Amazon Aurora Global Database
Automated endpoint management for Amazon Aurora Global Database

This sample code can be used to manage Aurora global database endpoints. After failover the global database writer endpoints swap from one region to the other. This solution automates creation and management of Route 53 based endpoints, so the applications don't have to change the connections strings.

Owner
James Jory
Applied AI Solutions Architect
James Jory
An attendance bot that joins google meet automatically according to schedule and marks present in the google meet.

Google-meet-self-attendance-bot An attendance bot which joins google meet automatically according to schedule and marks present in the google meet. I

Sarvesh Wadi 12 Sep 20, 2022
Python library for Spurwing API to schedule appointments, manage calendars and custom integrations.

Spurwing API Python Library Lightweight Python library for Spurwing's API. Spurwing's API makes it easy to add robust scheduling and booking to your a

Spurwing 1 Jul 14, 2021
An advanced Filter Bot with nearly unlimitted filters!

Unlimited Filter Bot ㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤ An advanced Filter Bot with nearly unlimitted filters! Features Nearly unlimited filters Supports all type of fil

TroJanzHEX 445 Jan 3, 2023
An advanced Filter Bot with nearly unlimitted filters!

Unlimited Filter Bot ㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤ An advanced Filter Bot with nearly unlimitted filters! Features Nearly unlimited filters Supports all type of fil

null 1 Nov 20, 2021
Filters to block and remove copycat-websites from DuckDuckGo and Google

uBlock Origin - Shitty Copy-Paste websites filter Filter for uBlock origin to remove spam-website results from DuckDuckGo and Google that just blatant

null 99 Dec 15, 2022
Filters to block and remove copycat-websites from DuckDuckGo and Google. Specific to dev websites like StackOverflow or GitHub.

uBlock-Origin-dev-filter Filters to block and remove copycat-websites from DuckDuckGo and Google. Specific to dev websites like StackOverflow or GitHu

null 1.7k Dec 30, 2022
An advanced Filter Bot with nearly unlimitted filters

Telegram MTProto API Framework for Python Documentation • Releases • Community Pyrogram from pyrogram import Client, filters app = Client("my_account

Pyrogram 3.2k Jan 5, 2023
PyFIR - Python implementations of Finite Impulse Response (FIR) filters

pyFIR Python implementations of Finite Impulse Response (FIR) filters. The algorithms are mainly (but not strictly) the ones described in WEFERS, Fran

Davi Carvalho 4 Feb 12, 2022
Easy to use phishing tool with 63 website templates. Author is not responsible for any misuse.

PyPhisher [+] Created By KasRoudra [+] Description : Ultimate phishing tool in python. Includes popular websites like facebook, twitter, instagram, gi

KasRoudra 1.1k Jan 1, 2023
Simple Telegram Bot for generating BalckPearl BBCode Templates

blackpearl-bbcode-bot Simple Telegram Bot for generating BlackPearl BBCode Templates Written in Pyrogram Features - ?? IMDB Info fetching from files -

D. Luffy 5 Oct 19, 2022