It may happen that an ETL task would send the same data more than once.
One scenario that would make this happen is resharding: a document can
be sent from one shard by the shard's ETL task, resharded, and then
sent again to the ETL destination by its new shard's ETL task.
Some ETL destinations will store duplicate incoming documents instead
of their former copies. Others, like OLAP and Queue ETL
destinations, will Not automatically recognize such events.
It is the user's responsibility to verify that the loaded documents
are handled as expected when they arrive.
OLAP helps users detect duplications using
lastModified, see a more
thorough discussion of this here
and relevant code samples here.