returned unexpected encoding "gzip" about s3backer HOT 12 CLOSED

This is very odd. When compression is enabled, s3backer uses content encoding deflate, not gzip. I have no idea how blocks were written with encoding gzip....

Are you or were you doing anything unusual?

from s3backer.

Nothing unusual. It seems like the issue occured as compression was enabled with blank password by shell script which was used.

from s3backer.

If this is reproducible, could you create a simple shell script or short test of some kind that demonstrates the problem, when run against a fresh/empty Amazon S3 bucket? Thanks.

from s3backer.

More info is available now. s3backer was tested in HCP ( hitachi ) and HCP stores the data in gzip format. Disabling the content-encoding header will save the data in HCP as it is. As per them, the latest unreleased version of HCP has the fix for this. Is content-encoding enabled by s3backer only when encryption is enabled ?

from s3backer.

Looks lik HCP is adding 'gzip' compression on the fly when sending the response. This is because s3backer inlucdes the Accept: */* header (actually it is libcurl that is adding this).

This looks like a SHOULD violation of the spec; see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3

Try this patch and let me know if this fixes the problem:

diff --git a/http_io.c b/http_io.c
index 0fa47cc..24cb016 100644
--- a/http_io.c
+++ b/http_io.c
@@ -37,6 +37,7 @@
 #define AUTH_HEADER                 "Authorization"
 #define CTYPE_HEADER                "Content-Type"
 #define CONTENT_ENCODING_HEADER     "Content-Encoding"
+#define ACCEPT_ENCODING_HEADER      "Accept-Encoding"
 #define ETAG_HEADER                 "ETag"
 #define CONTENT_ENCODING_DEFLATE    "deflate"
 #define CONTENT_ENCODING_ENCRYPT    "encrypt"
@@ -1022,6 +1023,7 @@ http_io_read_block(struct s3backer_store *const s3b, s3b_block_t block_num, void
     struct http_io_private *const priv = s3b->data;
     struct http_io_conf *const config = priv->config;
     char urlbuf[URL_BUF_SIZE(config)];
+    char accepted_encodings[64];
     const time_t now = time(NULL);
     int encrypted = 0;
     struct http_io io;
@@ -1088,6 +1090,14 @@ http_io_read_block(struct s3backer_store *const s3b, s3b_block_t block_num, void
         io.headers = http_io_add_header(io.headers, "%s: \"%s\"", header, md5buf);
     }

+    /* Set Accept-Encoding header */
+    snprintf(accepted_encodings, sizeof(accepted_encodings), "%s", CONTENT_ENCODING_DEFLATE);
+    if (config->encryption != NULL) {
+        snprintf(accepted_encodings + strlen(accepted_encodings), sizeof(accepted_encodings) - strlen(accepted_encodings),
+          ", %s-%s", CONTENT_ENCODING_ENCRYPT, config->encryption);
+    }
+    io.headers = http_io_add_header(io.headers, "%s: %s", ACCEPT_ENCODING_HEADER, accepted_encodings);
+
     /* Add Authorization header */
     if ((r = http_io_add_auth(priv, &io, now, NULL, 0)) != 0)
         goto fail;

from s3backer.

it sort of fixes the problem with the Hitachi Content platform however it now breaks large file transfers going to Amazon S3 for some reason

from s3backer.

Any more info on this? Are you still having problems with that patch and if so can you provide more details?

from s3backer.

We were able to work through this issue by implementing gzip support as a content encoding mechanism that s3backer can understand. We are not using the above proposed patch.

from s3backer.

Ok thanks.

Next question, now I'm wondering if this patch is OK to commit. Obviously you won't be using it in this case, but more generally it does seemingly fix a bug in s3backer.

However, your comment that:

it now breaks large file transfers going to Amazon S3 for some reason

worries me. Any idea what was going on there?

from s3backer.

I do not believe that issue was caused by this patch. I expect that it was only detected while testing with this patch. @eolson78 should have more history here and be able to clarify.

from s3backer.

OK good, that would make more sense.

from s3backer.

Patch applied in d0e863e.

from s3backer.