Downloading and Data Extraction Pipeline
For small data, the download could be a simple task. But in BigData regime, data download/transfer or server migration is a challenging task. One need to use command lines in shell scripts to transfer data from one remote server to another remote server. For example, sometimes data may be corrupted during the downloading process, an MD5-checksum need to be created and tested for integrity of the download process.