Architecture about pachyderm HOT 3 CLOSED

Good questions.

In the HDFS variant, the whole key for performance, if any, was: "move computation to the da
ta", which was achieved by maintaining multiple parts of a big file across machines as multiple copies.

How is this addressed in pfs - does it also maintain multiple copies of same file as multiple blocks / parts (if yes, how) ?

Pfs has a similar concept of bringing the computation to the data. Right now we don't do block storage for large files but we'll be implementing that in the near future. I've opened #58 to track that issue and there's a more in depth description of what the implementation will look like.

In the MapReduce variant of hadoop, for any given job, same copy of map-reduce method gets run on all machines, which makes it not suitable for data-flow computations, and which is not the case or concept for 'micro-services'. The core concept of micro-services is to build larger complex computation out of small and different disparate components /services. Which gives more control because you can tune / control individual micro-services without affecting the whole network.

I'm not 100% sure I understand what's being asked here so apologies if my answer doesn't address your concerns. Pachyderm organizes jobs in to pipelines, this gives it an understanding of where the data's next destination is which allows us to efficiently stream data between jobs.

from pachyderm.

Thanks @jdoliner Will look into the referred issues (such as #58) to get more info on this. Will get back in case of any questions.

from pachyderm.

Closing. @KrishnaPG If you have other questions, feel free ask on GH or email us [email protected].

from pachyderm.