Sometimes things go wrong. We worked with a client on a relatively straight forward ETL — extract, transform & load — engagement where we helped them extract data from their current system, transform it to fit a new data model and load it into another system. The ETL process was intended to run once, was completed, the result data reviewed / tested by the client, etc, and the new system went live.
Two days later it was discovered that the there was a significant problems with the data. The client had not realized the importance of some additional data, which hadn't been identified for ETL, and was critical... but that this point we can't just "go back" to the old system while the ETL process is updated, re-run and tested, the new system is live, is servicing new business, and must not be rolled back.
The resulting week has been frenzied, but we effectively pulled all data from the new system, combined it with the missing data from the old, and reloaded it — all without overloading the new system or any downtime.