Pega remove duplicates from activity and data transform

A Page List, such as a list of transactions loaded from two sources, sometimes contains the same item more than once. Pega gives you ready-made functions to remove the repeats, and you can use them from both a data transform and an activity.

The built-in functions

Pega includes functions for removing duplicates from a list page, for example RemoveDuplicatesInPageList and pzRemoveDuplicatesFromListPage. They take the page list and the property or properties that define a duplicate, and keep the first row for each value. Check the exact signature in the Function rule on your Pega version before you use it.

In a data transform

1. Add an action Set and set a local property, for example Param.Result, to the function call.
2. Pass the target list (for example .Transactions) and the key property (for example TransactionID).
3. Run the data transform and check the list on the clipboard.

In an activity

Use a Property-Set step and call the same function in the value field. The list on the step page is cleaned in place.

A worked example

A bank merges card transactions from two feeds into .Transactions. Some payments appear in both feeds, so a customer sees them twice. After the merge step, the data transform calls the remove-duplicates function on .Transactions using TransactionID as the key. The list drops from 128 rows to 121, and the screen shows each payment once.

If you prefer no function

You can do the same with plain rules:

1. Sort the list by the key property with Page-Sort or a report.
2. Loop through the list. If a row's key matches the previous row's key, remove the row with Page-Remove (go backwards through the list so the indexes stay correct).

Tips

• Decide which row to keep before removing duplicates, for example the newest. Sort first, then remove.
• Fix the source if you can. If a report causes the duplicates, tick Remove duplicate rows there instead. See how to remove duplicates in a Report Definition.
• Test with a list that has three copies of one item, not only two.

Related: the Data Transform label.