The first I tried to upload some TBs of data to a GCP bucket, I’ve received the following message:
NOTE: You are uploading one or more large file(s), which would run significantly faster if you
enable parallel composite uploads. This feature can be enabled by editing the "parallel_composite_upload_threshold"
value in your .boto configuration file.
Prerequisites
- GCP account
- gsutil
Solution
The default value I believe is set to 0 though. So, if you want to get rid of this message, you need to set this parameter value. For instance:
gsutil -o GSUtil:parallel_composite_upload_threshold=100M cp ./<some_subdir> gs://<your_bucket_name>/
or, in your $HOME/.boto file add the following line:
parallel_composite_upload_threshold = 100M
Conclusion
If you get stuck at some step, feel free to leave a comment below and if you find this tutorial useful, follow our official channel on Telegram.