​
Hi ​@andyzilla,

Thanks for your question!

At the moment, there isn’t a Pigment setting that will automatically drop duplicates and let the import continue. The best approach is either to remove duplicates before import or use a staging area in Pigment to identify and manage them.

Here’s how it works today:

If you enable Unique item values on a property, Pigment will stop the import if duplicates are found. However, it won’t filter or remove those duplicate rows on its own.

What you can do instead:

Clean up before importing
The smoothest option is to remove duplicates in your source system (HRIS) or by using a data-prep/ETL step before the data reaches Pigment. That way, your imports run successfully every time.

Handle duplicates inside Pigment
Another approach is to bring the data into a staging Transaction List (without unique constraints). From there, you can create a Metric or formula to flag rows with duplicate keys (e.g., Employee ID + Effective Date) so you can review and resolve them before moving the data into your main model.
 

Hope this helps! Let us know if you’d like more detail on setting up a staging list or duplicate flagging, we’re happy to walk you through it! 😊

​

Hi Tobias!

Thank you, this is helpful.

I’ll try handling the duplicates inside Pigment with a staging list, as I want to leverage the automation from my direct connectors. 

Quick update: I just ended up pre-processing the dataset in python.