In my first post I introduced you to the Sitecore to CDP module, which will create a structured extension with all Send email lists and if the Guest subscribed to them. In this post I'll walk through the process of getting all data out of chosen Send lists so they can be sent to CDP as a batch file.

Ok here's where we're at. You can see the 2nd stage in this process is to get the subscribers from Send. 

1. Module Overview, the Extension and Module Configuration
2. Download Users from Send Lists <- We're here
3. Cache and Date Stamps
4. Upload to CDP

Choosing Email Lists

When setting up the module, you create an Email List item for each list in Send you want to use. These email list items are an insert option under the main settings item found at /sitecore/system/Modules/SitecoreFundamentals/Send to CDP Sync/Send to CDP Sync Settings. All you need is the Name and ID of each email list.

Once created, go into Send to CDP Sync Settings to select which list you want to use.

Getting Email Lists Part 1 - Property Preparation

The Name field in the above email list items will dictate what the properties are going to be in the extension. As you can see here, each list creates two properties.

In reality the module creates the following properties in the JSON file (this is only part of it), but the CDP interface spaces and capitalizes them on its own:

"extensions": [
    {
        "name": "ext1",
        "key": "default",
        "subscriptionNewsletterSubscribers": "True",
        "subscriptionNewsletterSubscribersLastUpdated": "2025-09-01T14:11:07.2078183Z",
    }
]

You might ask, how are the list names changed to these property names? This is done in the main job right before Send is engaged. 

So far, we have a dictionary that represents email lists called EmailLists. Any that don't start with the word "subscription" gets it added before the next step.

var keysToUpdate = EmailLists.Keys.Where(k => !k.ToLowerInvariant().StartsWith("subscription")).ToList();
foreach (var key in keysToUpdate)
{
    var value = EmailLists[key];
    EmailLists.Remove(key);
    EmailLists[$"subscription{key}"] = value;
}

Getting Email Lists Part 2 - Processing the Lists

The next stage involves using the Send API to get all subscribers of the lists you've selected. This is done by calling ProcessMaillingListsAsync in a loop like this:

foreach (var emailList in EmailLists)
{
    var result = ProcessMaillingListsAsync(emailList).Result;
    if (!result)
    {
        Log.Error($"{logPrefix} {message}", this);
        return false;
    }
}

The method itself will make a call for each state the subscriber can be in, and consider only those who are Subscribed as, well, subscribed.

private async Task<bool> ProcessMaillingListsAsync(KeyValuePair<string, string> currentEmailList)
{
    using (var sendGateway = new Gateways.SendGateway())
    {
        var emailListMembers = await sendGateway.GetAllSubscribersOfMailingListAsync(currentEmailList.Value, MailingListMemberStatus.Subscribed);
        if (!ProcessMailingList(currentEmailList, emailListMembers.Context.Subscribers, true))
            return false;
        emailListMembers = await sendGateway.GetAllSubscribersOfMailingListAsync(currentEmailList.Value, MailingListMemberStatus.Unsubscribed);
        if (!ProcessMailingList(currentEmailList, emailListMembers.Context.Subscribers, false))
            return false;
        emailListMembers = await sendGateway.GetAllSubscribersOfMailingListAsync(currentEmailList.Value, MailingListMemberStatus.Removed);
        if (!ProcessMailingList(currentEmailList, emailListMembers.Context.Subscribers, false))
            return false;
        emailListMembers = await sendGateway.GetAllSubscribersOfMailingListAsync(currentEmailList.Value, MailingListMemberStatus.Bounced);
        if (!ProcessMailingList(currentEmailList, emailListMembers.Context.Subscribers, false))
            return false;
    }
    return true;
}

You can see that GetAllSubscribersOfMailingListAsync gets called here. This method will call GetPagedSubscribersOfMailingListAsync and check if there are more than 500 records in the result. If this is the case it will use the rate limit from the settings item (1 second) and paginate until complete. This is the part of the process that can take a while. In my case I have several hundred thousand subsribers. 

Building the Guest Record List

The above method calls ProcessMailingList, which has some important steps so I'm including it here.

This method will iterate over each record and perform a lookup in the NewCopyOfGuestRecords. It will create a new record of populate an existing one until all lists are complete.

Why "new" copy you ask? We'll get into that when we talk about caching. For now, know this private property is a Dictionary<string, Models.BatchUpload.GuestRecord>. A dictionary is used with the string being the identifier (email) instead of a list of guest records. Using a dictionary cuts this stage of the process down from about 2 hours in this case to a minute. 

private bool ProcessMailingList(KeyValuePair<string, string> currentEmailList, 
List<MailingListSubscriber> emailListMembers, 
bool isSubscribed)
{
    var logPrefix = $"[{GetType().FullName}.{MethodBase.GetCurrentMethod().Name}] ->";
    try
    {
        foreach (var emailListMember in emailListMembers)
        {
            var emailKey = emailListMember.Email.ToLowerInvariant();
            if (!NewCopyOfGuestRecords.TryGetValue(emailKey, out var guestRecord))
            {
                // Add new guest record if it doesn't exist
                guestRecord = new GuestRecord(emailListMember.Email);
                var guestExtension = GuestExtension(CdpExtensionName);
                guestExtension.Add(currentEmailList.Key, isSubscribed.ToString());
                guestExtension.Add($"{currentEmailList.Key}LastUpdated", DateTime.UtcNow.ToString("o"));
                foreach (var emailList in EmailLists.Where(x => x.Key != currentEmailList.Key))
                {
                    guestExtension.Add(emailList.Key, "false");
                    guestExtension.Add($"{emailList.Key}LastUpdated", DateTime.UtcNow.ToString("o"));
                }
                guestRecord.Value.Extensions.Add(guestExtension);
                NewCopyOfGuestRecords[emailKey] = guestRecord;
            }
            else
            {
                // Update existing guest record
                var extension = guestRecord.Value.Extensions
                    .FirstOrDefault(x => x.TryGetValue("name", out var name) 
                    && name == CdpExtensionName);
                if (extension == null)
                {
                    var guestExtension = GuestExtension(CdpExtensionName);
                    guestExtension.Add(currentEmailList.Key, isSubscribed.ToString());
                    guestExtension.Add($"{currentEmailList.Key}LastUpdated", DateTime.UtcNow.ToString("o"));
                    guestRecord.Value.Extensions.Add(guestExtension);
                }
                else
                {
                    if (extension.ContainsKey(currentEmailList.Key))
                    {
                        extension[currentEmailList.Key] = isSubscribed.ToString();
                    }
                    else
                    {
                        extension.Add(currentEmailList.Key, isSubscribed.ToString());
                    }
                }
            }
        }
    }
    catch (Exception ex)
    {
        Log.Error($"[{GetType().FullName}.{MethodBase.GetCurrentMethod().Name}] -> Error processing mailing list {currentEmailList.Key} ({currentEmailList.Value})", ex, this);
        TaskSummary.AppendLine($"Error processing mailing list {currentEmailList.Key} ({currentEmailList.Value}): {ex.Message}");
        return false;
    }
    var timeMessage = $"Processing list complete. {NewCopyOfGuestRecords.Count} total records so far.";
    TaskSummary.AppendLine(timeMessage);
    Log.Info($"{logPrefix} {timeMessage}", this);
    return true;
}

Once complete, we will have a list of Guests with an extension and true/false for each mailing list you selected.

What's Next?

Now that we have a comprehensive list of all subscribers / guests for upload, we want to cut down on the batch file size if we can. The local caching mechanism will handle this so only changes are deployed. Stay tuned!