HDFS calculates a CRC over each block and combines them into a file-level checksum. S3 has no equivalent. An ETag on a single-part upload is an MD5 of the object; on a multipart upload it is an MD5 of the part MD5s followed by the part count. Neither can be compared with an HDFS checksum, so distcp either skips the comparison for S3 targets or compares against a value it computed on the source side only. A successful job therefore tells you that every copy operation returned without error, not that the bytes on both sides match.

Mismatches usually come from three places. A source file changed between the listing phase and the copy phase, so what landed on S3 is a snapshot of a file that no longer looks like that. A multipart upload was interrupted and retried, and the retry produced a valid object whose ETag no longer reflects the original part boundaries. Or the job was run with checksum checking disabled to get past an earlier failure, and nobody turned it back on.

The practical steps are to run distcp with the S3A committer configured for the store you are writing to, to keep the source quiet during the copy or accept that live paths need a second pass, and to verify after the fact by recomputing a content hash on both sides rather than trusting ETags. For estates that cannot be frozen, a tool that tracks changes as they happen and reconciles continuously removes the listing-to-copy gap that causes most of these mismatches in the first place.

A lift-and-shift to CDP Public Cloud keeps your tooling, security model and operational habits intact. It buys time and reduces the risk of breaking pipelines that nobody fully understands any more. The cost is that you carry the licence and the platform dependency forward, and you are still running a distribution designed around HDFS on infrastructure where object storage is the native layer.

Leaving the platform entirely means landing the data on S3, ADLS or GCS and running compute on a lakehouse engine. That is where most new investment is going and where the AI platforms expect to find data. The cost is rewriting, or at least re-testing, everything that assumed HDFS semantics, Hive on YARN, or Impala.

The pattern we see working is to separate the two decisions. Move the data first, continuously, so that a copy of the estate exists in object storage while production keeps running on the old cluster. Then move workloads one at a time, choosing per pipeline whether it goes to a CDP service or to a lakehouse engine, and validating each against the live copy rather than a stale export. The data move stops being the blocker, and the compute decision can be made on the merits of each workload rather than under a support deadline.

Every table definition carries a storage location. On the old cluster that is an hdfs:// path; on the target it needs to be an s3a://, abfs:// or gs:// path, and every partition under the table has its own location that needs the same translation. Exporting the metastore and re-importing it without rewriting those paths produces tables that exist but point at nothing.

Version drift is the second problem. The on-prem Hive metastore schema and the target service (a cloud-managed Hive-compatible catalog, Glue, or an Iceberg catalog) rarely match. Serde definitions, table properties and statistics that were valid on one side are silently dropped or rejected on the other. Statistics in particular tend to be stale after the move, so query plans degrade until they are recomputed.

The third problem is time. If the migration runs over weeks or months, the source metastore keeps changing. New partitions land daily; tables get altered. A one-time export is out of date by the time it is loaded. The approach that works is to treat metadata as a stream, not a snapshot: capture changes to the source metastore as they happen, translate locations and types on the way through, and apply them to the target so that the two stay in step until cut-over. Files first, then the metadata that references them, so that a table never appears on the target before the data it points to.

Rename is the one that hurts most. On HDFS, moving a directory is a metadata operation that completes instantly and atomically. On object storage it is a copy of every object followed by a delete, which is slow and can be observed half-done. Any job that writes to a temporary location and renames on completion, which is most classic MapReduce and older Spark, becomes both slower and unsafe. The mitigation is to use a committer designed for object storage rather than the file-output committer.

Directories do not exist in an object store; they are inferred from key prefixes. Code that checks whether a directory exists, creates empty directories as markers, or relies on directory modification times will behave differently. Listing large prefixes is slow and paginated, and on some stores a freshly written object may not appear in a listing immediately.

Small files are cheap on HDFS and expensive on object storage, because every object carries a per-request cost and a listing overhead. Pipelines that produce thousands of small partitions per day need compaction added. Finally, permissions change shape: HDFS ACLs and Ranger policies do not map directly onto bucket policies and IAM. None of this is a reason not to move, but it is a reason to test each workload against the target store before it is cut over, ideally against a live copy of the real data rather than a sample.

The problem with a batch copy is that its listing and its copying happen at different times. Files created after the listing are missed. Files modified during the copy are captured in whatever state they were in when the copier reached them. A checksum comparison at the end will show differences, and the honest answer to "which side is right" is that neither reflects a single point in time.

Freezing the source solves this but is rarely acceptable for a production estate. The alternative is to capture every change on the source as it happens, apply it to the target in order, and keep a record of what has been applied. At any moment you can then state which source events the target reflects. At cut-over you stop writes for a short window, let the last events drain, and reconcile.

What an auditor wants is that reconciliation, at a named timestamp, showing that for every path on the source there is a matching path on the target with a matching content hash, and listing any exceptions with a reason. Produce it as a report, keep it with the migration record, and make sure it was generated by comparing actual content on both sides rather than by trusting the copy tool's own success log. The report is the evidence; the continuous tracking is what makes it possible to generate one that is true.

Data Migrator needs to be able to read HDFS as a client and to talk to the NameNode, which means it sits on a host that has the Hadoop client libraries, network access to the cluster and appropriate Kerberos credentials. It does not need to run on the data nodes and does not modify the cluster. Most deployments put it on a dedicated edge node sized for the outbound bandwidth to the target.

Once a migration is running, ongoing changes are picked up from the NameNode's change notifications. That is a lightweight subscription rather than a repeated directory walk, so the load it adds to the NameNode in steady state is small and predictable. The initial pass is different: to establish the baseline it has to enumerate the paths in scope, and on a very large namespace that enumeration is a real read load. Plan it for a quiet period, scope it by path rather than migrating the root, and expect the data nodes to be busy serving reads while the initial transfer runs.

The other capacity question is outbound network. Data Migrator will use whatever bandwidth it is given, so set a limit if the link is shared with production traffic.

Start with the arithmetic. If the copy runs at midnight and takes four hours, a failure at 23:00 loses everything since the previous midnight, and the copy that exists reflects files as they were somewhere between midnight and 04:00. That is not a recovery point; it is a range. Anything that depends on two files being in step, which is most table data plus its metadata, may be in a state that never existed on the source.

Continuous replication captures each change as it happens and applies it to the second site in order. The recovery point becomes the replication lag, typically seconds to minutes depending on network and change rate. That is the headline benefit. The costs are real: the link between sites carries every change, not a batch, so bandwidth needs to be provisioned for peak write rate rather than average; deletes have to be replicated too, and you need a policy on whether the DR copy should mirror deletes immediately or retain them; and a large one-off rewrite on the source will show up as a burst on the wire.

The question to ask before choosing is what the business actually needs. If the data lake feeds overnight batch and a day's loss is tolerable, a nightly copy with a proper reconciliation may be enough. If it feeds anything that runs during the day, the range-not-point problem is the thing to fix first.

"Provably identical at any point in time" is stronger than it sounds. It means that for any timestamp the regulator picks, you can say what the target contained and show that it matched the source as of that moment. A copy tool that scans the source periodically cannot do this; it can only say that at the end of the last scan the two sides matched, and it cannot say anything about the interval between scans.

Event-based replication can, because it works from an ordered stream of changes on the source. Each applied event is recorded, so the target's state at any moment is defined as the source's state as of the last event applied. A reconciliation at a chosen time is then a content comparison of every path on both sides, with the position in the event stream recorded alongside. Keep those reports as part of the compliance record.

Cross-region and HDFS-to-object-storage add two complications. Latency between regions widens the replication lag, so the "as of" point on the target trails the source by more; state that lag explicitly. And because the target is an object store, content verification has to use a hash you computed rather than the store's ETag, for the reasons covered in a previous question (see the distcp checksum answer in this archive). Build the verification into the replication, not as a separate job that runs afterwards.

The data files are the easier half. Parquet or ORC files under a Hive table's location can be replicated to object storage as they are written, and both targets can read them from there. Replicating once to a shared object store and pointing both catalogs at it avoids maintaining two copies of the data and keeps the two platforms reading the same bytes.

The metadata is the harder half. Hive tracks partitions as directories; Iceberg tracks table state as a sequence of snapshots, each listing the exact data files it includes. Translating one to the other means generating Iceberg manifests and snapshot metadata that reference the replicated files, then committing those to each target catalog. If the source table gains a partition, the target needs a new snapshot that includes it.

The ordering rule is that a snapshot must never be committed on a target before every file it references has landed. Do it the other way round and readers on the target see a table pointing at objects that do not exist yet, and queries fail or, worse, return partial results. So the pipeline has to confirm file arrival before it publishes metadata, per snapshot, on each target.

Production does not need to freeze because nothing here requires the source to stop. What it requires is that the replication captures changes in order and that the metadata translation is driven by the same event stream, so that both targets converge to the same state at slightly different times rather than diverging.

Egress is the visible cost. Every time a model training run or a feature pipeline pulls data out of the platform it lives on, someone pays per gigabyte, and AI workloads read the same data repeatedly. Latency is the less visible one: a training job reading across a region boundary or through a slow gateway spends its expensive GPU time waiting. And copies are the cost that compounds, because each team that cannot get at the data where it is makes its own extract, and now there are five versions with no owner.

Moving compute to the data works when there is one consumer, the job runs once or rarely, and the platform where the data lives can host the compute. If the data sits on an on-prem Hadoop cluster and the AI platform is in the cloud, that condition usually fails; the cluster cannot run the workload and cannot take the extra read load on top of production.

Moving data works when more than one platform needs it, when the consumers run continuously, or when the source cannot be loaded further. The condition is that the moved copy stays current, otherwise you have simply created another stale extract. Count the consumers and the frequency. One consumer, one run: move compute. Several consumers, continuous use: move the data once, continuously, to a location all of them can read, and stop making copies.

On Hadoop, the things that can be lost are the NameNode's view of the filesystem, the blocks themselves, and the running services. DR means a second cluster with the data replicated to it, the metastore and security policies mirrored, and a runbook for repointing clients. The DR test is to stand up the second cluster and run the workloads.

On object storage, durability of the objects is the provider's problem, and within a region it is very high. What the provider does not protect is availability across a regional outage, the correctness of what your pipelines wrote, or your catalog. The realistic failure modes are a region becoming unreachable, a bad deployment deleting or corrupting a table, and a catalog that gets out of step with the files. The protection is a second copy of the data in another region or cloud, a second copy of the catalog, and a known lag between them.

"Failover" therefore means repointing compute engines at the second copy of the data and the second catalog, and the DR test is whether the engines can query the second copy and get the same answers. That test is only meaningful if the second copy is current, which brings you back to replication lag as the number that defines your recovery point.