S3: get/put fail when multi-part file already on S3

[lorrin]

The s3 module has a keysum method that fails when the file on S3 got there via multipart upload:

etag_multipart = md5_remote.find('-')!=-1 #Check for multipart, etag is not md5
if etag_multipart is True:
    module.fail_json(msg="Files uploaded with multipart of s3 are not supported with checksum, unable to compute checksum.")

It is used by get and put in overwrite scenarios to determine whether the source and destination are identical. The resulting behavior is that the module fails prematurely, regardless of whether overwrite=True was passed.

[mpdehaan]

Can you please share the output of "ansible --version" and also share some examples that show with what you are doing in the playbook in addition to just showing the internal code snippets?

Thanks!

[lorrin]

I'm running ansible 1.4.1, but the bug appears to still be present in master.

The relevant part of the playbook is:

- name: Fetch Spark
  s3: bucket=my-bucket object=/spark-0.8.1-incubating-bin-hadoop1.tgz dest=/tmp/spark-0.8.1-incubating-bin-hadoop1.tgz mode=get aws_access_key={{aws_access_key_id}} aws_secret_key={{aws_secret_access_key}}

As a successful work-around, I re-uploaded the file to S3 as a single-part upload, which made the eTag be an md5.

[lorrin]

@mpdehaan, let me know if there is more info you need. Cheers.

[jimi-c]

See my recent comment on the mailing list in regards to multipart handling with the s3 module. Unfortunately, there's no simple solution so for the time being the solution is as you've done above. If you have any questions about this, please let us know. Thanks!

[hiwaylon]

@lorrin How were you able to re-upload the file as a single-part upload? I cannot find any details at all on how to prevent multipart uploading.

[lorrin]

@hiwaylon, IIRC I used s3cmd rather than the standard AWS CLI to do the upload.

[willybarro]

If anyone else is having problem with that and don't really care of downloading the file again, a possible workaround is to remove the object from the target before trying to redownload from S3. Something like this:

- name: Remove artifact if it exists locally.
  file: path=/tmp/file.deb state=absent

- name: Download artifact from S3
  s3: aws_access_key={{ s3_access_key }}
      aws_secret_key={{ s3_secret_key }}
      bucket={{ s3_bucket }}
      object=file.deb
      dest=/tmp/file.deb mode=get

[lindes]

It would be nice if there were some way to have this work... even if it didn't check the checksum, and/or checked a user-supplied checksum (see ansible/ansible-modules-core#2404 )...

[iainelder]

I don't understand why this is closed without a fix.

The native aws s3 cp handles multipart objects just fine.

Can Ansible not just use that?

[jimi-c]

See my post here (https://groups.google.com/d/msg/ansible-project/Dg7aAmpoS2A/iWkJ6Yp2L3YJ) for the reasons why I closed this. We use boto internally, and have no plans on changing that.