What if you wanted to instantly see all Send email lists your CDP Guests are subscribed to? This module will collect information from Send and populate CDP with what they've signed up for. It also maintains the last updated timestamps for each list preference and deploys the changes in batch files. Uploads are minimal by using a local cache, and the preferences are structured in a single extension.

I haven't posted in a while because I've been working on something bigger than usual. As part of a larger integrations project, I needed to collect all Send preferences for guests (identified by email), and have their information show up in CDP as an extension. There are ways you might be able to accomplish this with established tools, but after meeting with Send SMEs, we agreed a module is the most cost-effective way to proceed since there would be a massive drop in API calls by leveraging the batch functionality of CDP.

This is part one of a four-post series since there's a lot to cover. They will be: 

1. Module Overview, the Extension and Module Configuration
2. Download Users from Send Lists
3. Cache and Date Stamps
4. Upload to CDP

The Module

Before we get started, you can download the module from my GitHub account. I'll be referring to the files there since it's too big to paste ad-hoc content throughout the coming posts.

The Extension

If you look at the Guest extension data model in CDP, you'll see you can add custom properties to any guest. Name and Key are required of each extension, and what's really important to remember for later is, when you deploy changes, you need to include all of your extension's data each time you want to update a guest. It's for this reason my module cache's the timestamps for changed properties, since we have to rebuild the entire extension each time a change is made (Send doesn't keep this timestamp on record or available in the API).

The module has a BatchUpload class under Models, which outlines the structure of the file which is generated for each upload. In it, there's a list of GuestRecords, which has a child called GuestValues. This class has the extensions area, which is created using the following method. As you can see, the two mandatory properties of Name and Key are generated automatically each time a new object is made. 

public static Dictionary<string, string> GuestExtension(string extensionName)
{
    return new Dictionary<string, string>
    {
        { "name", extensionName },
        { "key", "default" }
    };
}

Module Configuration

This module is driven by one config file, a main settings item, which will have child items (as insert options) which represents each email list in Send you want to include.

The Config File

The config item has settings for the folder where cache files will be generated, which extension should be created (in this case ext1), and if deploying to CDP should be stopped in case of local development is underway.

The config also has the agent, which runs the pipeline every six hours. You can adjust this frequency, but I can tell you most of the work is related to getting Users from Send. In my case I have about 350,000 Users across a few lists, and at a Send API rate limit of 1 second for every 500 Users, that's about 12 minutes. The entire job is about 14 minutes to complete in this case.

The Settings Item

The module comes with a settings item which lets you enter credentials for both Send and CDP, and choose which email lists to use in your Send instance. 

In the first data section you choose to turn on or off the service, and which email lists to use. Modifying the chosen Send lists will require a full upload in your next batch file since CDP needs the complete extension each time.

The email lists are children of the settings item, and only need two fields. The name field will become the Boolean property in the CDP extension.

The next data section in the main settings item is a password needed for encrypting the local cache files. Though they're in the app_data folder, they will be zipped with a password.

The Job Statistic data section is self explanatory. It will show the last run's log and the time of the last and next run.

The next two sections are used to set up the connection to Send and CDP. There are standard values for all you see here, so all that's needed are your API keys and token. Once these are set, and you've entered the email lists you need (above), you're good to go.

Your First Run

I recommend disabling the upload to CDP by using the Boolean value in the config until you've had a chance to look at the batch file manually. I think it's a good testing measure before committing anything to your account. We'll cover these steps in the next posts, but for now, install it and get everything ready. Good luck!