<p>Let me first say that being able to take 17 million records from a flat file, pushing to a DB on a remote box and having it take 7 minutes is amazing. SSIS truly is fantastic. But now that I have that data up there, how do I remove duplicates?</p>

<p>Better yet, I want to take the flat file, remove the duplicates from the flat file and put them back into another flat file.</p>

<p>I am thinking about a:</p>

<p><strong><code>Data Flow Task</code></strong></p>

<ul>
<li>File source (with an associated file connection)</li>
<li>A for loop container</li>
<li>A script container that contains some logic to tell if another row exists</li>
</ul>

<p>Thak you, and everyone on this site is incredibly knowledgeable.</p>

<p><strong><code>Update:</code></strong> <a href="http://rafael-salas.blogspot.com/2007/04/remove-duplicates-using-t-sql-rank.html" rel="noreferrer">I have found this link, might help in answering this question</a></p>

I would suggest using SSIS to copy the records to a temporary table, then create a task that uses Select Distinct or Rank depending on your situation to select the duplicates which would funnel them to a flat file and delete them from the temporary table.  The last step would be to copy the records from the temporary table into the destination table.
Determining a duplicate is something SQL is good at but a flat file is not as well suited for.  In the case you proposed, the script container would load a row and then would have to compare it against 17 million records, then load the next row and repeat...The performance might not be all that great.