Bulk data loader
Diagram of the Bulk data loader system: 5 steps
Steps
- Client export: The client sends a spreadsheet export in whatever shape it comes.
- Maps the columns: The columns are matched to the fields the product expects.
- Rows that fail validation: Rows with missing or odd values are set aside for a look.
- Product API: Clean rows are written into the product in batches.
- Imported, with a report: The client gets a report of what went in and what did not.
25 hours of manual entry per client dataset, now minutes
The problem
At a software company, every new client arrived with their data in a spreadsheet: hundreds to a few thousand rows that had to end up inside the product. The product had no import, so an account manager typed the records in one by one through the interface, at 1 to 2 minutes per record. For a typical client that came to around 25 hours of entry, spread across the first weeks of the relationship, with the typos you would expect from hour 20. It was done by hand because nobody had built the alternative, and the client's first impression of the product was a wait.
The system
The loader takes the client's Excel export and maps its columns onto the product's data model. Claude proposes the mapping (which column is the name, which the date, which the unit) and a person confirms it once per client. Every row is then validated against the same rules the product applies in its interface: required fields, allowed values, dates in range. Rows that pass are sent to the product's API in batches. Rows that fail are written to a report with the reason, and nothing from that file is imported until a person has read the report. The import is create-only: it never overwrites or deletes an existing record.
What it does now
A dataset that took around 25 hours to type in now loads in minutes of machine time, plus the time a person needs to confirm the column mapping and read the error report, usually well under an hour. Validation errors are caught before the import instead of being found inside the product weeks later. A new client dataset is now a mapping session and an error report, not a week of typing.
Where AI checks and where it decides
| Step | Who decides |
|---|---|
| The proposed column mapping, and which rows fail validation and why | AI decides |
| Confirming the mapping, reviewing the failed rows, starting the import | A person decides |