---
title: 'How to Find and Remove Duplicates in Airtable'
description: 'Three methods to find and remove duplicate records in Airtable — the Dedupe extension, formula-based detection, and scripting for bulk cleanup.'
canonical_url: 'https://www.business-automated.com/tutorials/airtable-find-remove-duplicates'
md_url: 'https://www.business-automated.com/tutorials/airtable-find-remove-duplicates.md'
last_updated: 2026-09-30
---

Duplicates are the most common data-quality problem in [Airtable](/airtable-consultant) bases. They creep in from sync scripts that don't use unique keys, form submissions resubmitted by users, imports without dedup logic, and manual entry without enforcement.

This guide covers the three real methods to find and remove them, plus the patterns that prevent future duplicates from accumulating.

## Method 1: The Dedupe Extension (Fastest for One-Off Cleanups)

Airtable's built-in Dedupe extension is the fastest path to a clean table.

### Setup

1. In your base, open the **Extensions** panel.
2. Click **+ Add an extension** → search **Dedupe** → install.
3. Pick the table you want to dedupe.
4. Choose one or more fields to match on:
   - **Exact match** for IDs, emails (after normalization).
   - **Fuzzy match** for company names, person names — catches "Acme Inc." vs "Acme, Inc".
5. Click **Find duplicates**.

The extension groups records that match and shows each group for review.

### Review and resolve

For each duplicate group:

- **Pick a "primary" record** — the one to keep. Use built-in rules like "Keep the oldest" or "Keep the one with the most filled fields," or pick manually.
- The remaining records are marked for deletion.

Click **Delete** to remove the duplicates. The extension processes 100–500 records per minute depending on table complexity.

### Strengths and limits

**Good for:**
- One-time cleanups of up to 10,000 records.
- Tables where you want to review groups before deleting.
- Fuzzy matching across name variations.

**Not good for:**
- Tables over 50,000 rows (slow).
- Ongoing prevention (it's a one-time tool, not a continuous one).
- Headless / scripted cleanup.

## Method 2: Formula-Based Detection (Best for Prevention)

For continuous duplicate detection without manual review, build a formula that flags duplicates as they're created.

### Pattern: Duplicate Count rollup

The cleanest version uses Airtable's self-link pattern:

1. **Create a "Dedup Key" formula field.** Normalize the key — lowercase, strip whitespace:
   ```javascript
   LOWER(TRIM({Email}))
   ```

2. **Add a linked record field linking to the same table** (the table self-links). Call it "Self Link."

3. **Use an automation** to populate Self Link based on Dedup Key:
   - Trigger: When record created.
   - Action: Find records where Dedup Key = the new record's Dedup Key.
   - Action: Update the new record, linking Self Link to all matching records.

4. **Add a Count rollup field** that counts the linked Self Link records.

5. **Filter to Count > 1** to see all duplicate sets.

### Simpler alternative: pure formula flag

For a basic "is this a duplicate of a known good record" pattern, maintain a separate Allowlist table and use a lookup to flag duplicates. This is lighter-weight but only catches duplicates against the allowlist.

### Strengths and limits

**Good for:**
- Continuous detection — formulas update on every change.
- Surfacing duplicates in views without manual scans.
- Pairing with automations that handle duplicates at creation time.

**Not good for:**
- Bulk cleanup of existing duplicates (formulas detect; they don't delete).
- Fuzzy matching (formulas do exact match on the normalized key).

## Method 3: Scripting for Bulk Cleanup

For large-scale dedup or programmatic cleanup, Airtable's Scripting extension is the right tool.

### The script pattern

```javascript
const table = base.getTable('Contacts');
const records = await table.selectRecordsAsync({fields: ['Email']});

const seen = new Map();
const toDelete = [];

for (const record of records.records) {
    const email = (record.getCellValueAsString('Email') || '').toLowerCase().trim();
    if (!email) continue;
    
    if (seen.has(email)) {
        // Keep the first one we saw; delete this one
        toDelete.push(record.id);
    } else {
        seen.set(email, record.id);
    }
}

output.markdown(`Found ${toDelete.length} duplicates to delete.`);

// Confirm before deleting
const confirmed = await input.buttonsAsync(
    `Delete ${toDelete.length} duplicates?`,
    [{label: 'Cancel', value: false}, {label: 'Delete', value: true, variant: 'danger'}]
);

if (confirmed) {
    // Batch delete in groups of 50 (Scripting's max)
    while (toDelete.length > 0) {
        const batch = toDelete.splice(0, 50);
        await table.deleteRecordsAsync(batch);
    }
    output.markdown('Done.');
}
```

### Variations

- **Keep the most complete record:** Instead of keeping the first one seen, keep the one with the most filled fields by counting non-empty values.
- **Merge before delete:** For each duplicate group, copy non-empty fields from duplicates to the primary record before deleting.
- **Composite key:** Dedup by email + phone, not just email. Build the key as `LOWER(email) + '|' + phone`.

### Strengths and limits

**Good for:**
- Tables with 10,000–100,000 records.
- Custom dedup logic (composite keys, merge logic).
- Repeatable cleanup that can be run on demand.

**Not good for:**
- Truly fuzzy matching (write that in Python with pyairtable instead).
- Tables over 100K rows (Scripting times out — use the API from Python).

## Prevention: Stopping Duplicates Before They Happen

The best dedup is the one you never have to run. Three patterns prevent future duplicates.

### Pattern A: Upsert at the API layer

For any sync script or automation creating records, use Airtable's native upsert endpoint:

```python
table.batch_upsert(
    records=[{'fields': {'Email': 'a@b.com', 'Name': 'Acme'}}],
    key_fields=['Email']
)
```

If a record exists with the matching Email, it's updated. If not, created. Duplicates are impossible by construction.

See our [Airtable API guide](/tutorials/airtable-api-beginners-guide) and [Python guide](/tutorials/airtable-python-guide).

### Pattern B: Form submission dedup

For inbound forms, build an automation:

1. Trigger: New form submission.
2. Find records: Search the destination table for matching Email.
3. Conditional:
   - If found → Update the existing record with the new data.
   - If not found → Create a new record.

### Pattern C: Manual entry validation

For user-entered records, add a validation step:

1. Trigger: New record created.
2. Find duplicates by Dedup Key.
3. If duplicates found → Notify the user, optionally auto-merge or auto-delete.

This is reactive (the duplicate exists briefly), but better than letting them accumulate.

## Comparison: Method Decision Matrix

| Use Case | Best Method |
| --- | --- |
| One-off cleanup, under 10K records | Dedupe extension |
| One-off cleanup, 10K–100K records | Scripting |
| One-off cleanup, over 100K records | Python with pyairtable |
| Continuous detection in views | Formula-based |
| Prevent at API integration | Upsert with key_fields |
| Prevent on form submission | Automation with Find + Conditional |
| Fuzzy matching across name variations | Dedupe extension (basic) or Python + rapidfuzz (full) |

## Common Mistakes

**Mistake 1: Dedup without backing up.** Export the table to CSV before running any deletion. Mistakes happen; backups recover them.

**Mistake 2: Wrong dedup key.** Deduping contacts by Name catches "John Smith" vs "John Smith" but also collapses two real people. Use stable IDs (email, phone, external ID) instead.

**Mistake 3: Not normalizing the key.** "a@b.com" and "A@B.com" don't match without lowercasing. " a@b.com " doesn't match without trimming.

**Mistake 4: Letting duplicates re-accumulate.** Cleanup is half the work — implement prevention (upserts, form dedup) immediately after.

**Mistake 5: Trusting fuzzy matching unconditionally.** Fuzzy match catches real duplicates and false positives. Always review before bulk delete.

## Troubleshooting

**Dedupe extension finds zero duplicates but you can see them.** Match field has hidden differences (whitespace, capitalization). Switch to fuzzy match or normalize before running.

**Formula Count rollup returns 1 for everything.** Self-link automation isn't running or isn't populated. Confirm Self Link field has values.

**Script times out.** Table too large for Scripting (over 100K records). Move to Python with pyairtable.

**After dedup, linked records lost.** Deleted records had linked references that are now broken. Reassign links from duplicates to the kept record before deleting.

**Re-import re-creates duplicates.** Source CSV/sync doesn't use a dedup key. Add one and use upserts.

## Next Steps

Dedup is a one-time fix, but the patterns that prevent it are forever. Once your base is clean, lock in prevention via upserts at every entry point — API integrations, form submissions, automations creating records. The maintenance load drops to near-zero.

For broader patterns, see our [Airtable API guide](/tutorials/airtable-api-beginners-guide), [Python guide](/tutorials/airtable-python-guide), [scripting guide](/tutorials/airtable-scripting-guide), and [CSV import guide](/tutorials/how-to-import-csv-excel-into-airtable). For a one-time large-scale dedup or migration cleanup, [get in touch](/contact).


## Sitemap

See the full [sitemap](/sitemap.md) for all pages.