Comments (12)
jamesbornholt
commented on May 29, 2024
Hmm, I would have expected getting credentials from IMDS to work here, which the CRT (like all SDKs) does automatically. At least, this has worked for me in my testing on EC2 instances.
from mountpoint-s3.
jamesbornholt
commented on May 29, 2024
You could try running env RUST_LOG=debug s3-file-connector $HOME/mnt your_bucket_name to see the CRT's logs about auth.
from mountpoint-s3.
monthonk
commented on May 29, 2024
It works on EC2 instances but not the containers inside it.
CRT's logs give me this
2022-10-18T18:31:48.141299Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/home/fuseuser/.aws/config"
2022-10-18T18:31:48.141330Z ERROR awscrt::common-io: static: Failed to open file /home/fuseuser/.aws/config with errno 2
2022-10-18T18:31:48.141337Z WARN awscrt::AWSProfile: Failed to read file at "/home/fuseuser/.aws/config"
2022-10-18T18:31:48.141354Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/home/fuseuser/.aws/credentials"
2022-10-18T18:31:48.141375Z ERROR awscrt::common-io: static: Failed to open file /home/fuseuser/.aws/credentials with errno 2
2022-10-18T18:31:48.141392Z WARN awscrt::AWSProfile: Failed to read file at "/home/fuseuser/.aws/credentials"
2022-10-18T18:31:48.141401Z ERROR awscrt::AuthCredentialsProvider: static: Profile credentials parser could not load or parse a credentials or config file.
2022-10-18T18:31:48.141416Z DEBUG awscrt::AWSProfile: Creating profile collection from file at "/home/fuseuser/.aws/config"
2022-10-18T18:31:48.141434Z ERROR awscrt::common-io: static: Failed to open file /home/fuseuser/.aws/config with errno 2
2022-10-18T18:31:48.141445Z WARN awscrt::AWSProfile: Failed to read file at "/home/fuseuser/.aws/config"
2022-10-18T18:31:48.141462Z ERROR awscrt::AuthCredentialsProvider: Failed to build config profile collection from file at (/home/fuseuser/.aws/config) : Invalid file path.
2022-10-18T18:31:48.141474Z ERROR awscrt::AuthCredentialsProvider: Failed to resolve either region, role arn or token file path during sts web identity provider initialization.
2022-10-18T18:31:48.141489Z INFO awscrt::connection-manager: id=0x564cf1a773b0: Successfully created
2022-10-18T18:31:48.141507Z INFO awscrt::exp-backoff-strategy: id=0x564cf1a7dd70: Initializing exponential backoff retry strategy with scale factor: 0 jitter mode: 0 and max retries 1
...
2022-10-18T18:31:48.153857Z INFO awscrt::S3ClientStats: id=0x564cf1bd8d80 Requests-in-flight(approx/exact):1/1 Requests-preparing:1 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming:0 Endpoints(in-table/allocated):1/1
2022-10-18T18:31:48.153896Z DEBUG awscrt::task-scheduler: id=0x7f0bf0001098: Scheduling s3_meta_request_prepare_request_task task for immediate execution
2022-10-18T18:31:48.153917Z DEBUG awscrt::task-scheduler: id=0x7f0bf0001098: Running s3_meta_request_prepare_request_task task with <Running> status
2022-10-18T18:31:48.153949Z DEBUG awscrt::S3MetaRequest: id=0x564cf1bde040: Meta Request prepared request 0x7f0bf0000f20
2022-10-18T18:31:48.154060Z DEBUG awscrt::S3MetaRequest: id=0x564cf1bde040 Created signable 0x7f0bf4000c10 for request 0x7f0bf0000f20 with message 0x7f0bf4000b60
2022-10-18T18:31:48.154088Z DEBUG awscrt::AuthCredentialsProvider: (id=0x564cf1a730e0) Credentials provider chain get credentials dispatch
2022-10-18T18:31:48.154116Z INFO awscrt::AuthCredentialsProvider: (id=0x564cf1a77620) Cached credentials provider has expired credentials. Requerying.
2022-10-18T18:31:48.154132Z DEBUG awscrt::AuthCredentialsProvider: (id=0x564cf1a72410) Credentials provider chain get credentials dispatch
2022-10-18T18:31:48.154161Z DEBUG awscrt::AuthCredentialsProvider: (id=0x564cf1a72410) Credentials provider chain callback 1 invoked with invalid credentials and error code 6150
2022-10-18T18:31:48.154178Z DEBUG awscrt::AuthCredentialsProvider: (id=0x564cf1a72410) Credentials provider chain invoking chain member #1
2022-10-18T18:31:48.154212Z DEBUG awscrt::exp-backoff-strategy: id=0x564cf1a7dd70: Initializing retry token 0x7f0bf4003d70
2022-10-18T18:31:48.154252Z DEBUG awscrt::Unknown: (id=0x564cf1a772b0) IMDS client's token is invalid and is now updating.
2022-10-18T18:31:48.154273Z DEBUG awscrt::task-scheduler: id=0x7f0bf4003dc8: Scheduling aws_exponential_backoff_retry_task task for immediate execution
2022-10-18T18:31:48.154299Z DEBUG awscrt::task-scheduler: id=0x7f0bf4003dc8: Running aws_exponential_backoff_retry_task task with <Running> status
2022-10-18T18:31:48.154310Z DEBUG awscrt::exp-backoff-strategy: id=0x564cf1a7dd70: Vending retry_token 0x7f0bf4003d70
2022-10-18T18:31:48.154329Z DEBUG awscrt::Unknown: id=0x564cf1a772b0: IMDS Client successfully acquired retry token.
2022-10-18T18:31:48.154337Z DEBUG awscrt::connection-manager: id=0x564cf1a773b0: Acquire connection
2022-10-18T18:31:48.154349Z DEBUG awscrt::connection-manager: id=0x564cf1a773b0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=1, pending_settings_count=0, pending_connect_count=1, vended_connection_count=0, open_connection_count=0, ref_count=1
2022-10-18T18:31:48.154366Z INFO awscrt::connection-manager: id=0x564cf1a773b0: Requesting 1 new connections from http
2022-10-18T18:31:48.154378Z DEBUG awscrt::dns: id=0x564cf1a72310: Host resolution requested for 169.254.169.254
2022-10-18T18:31:48.154387Z DEBUG awscrt::dns: id=0x564cf1a72310: No cached entries found for 169.254.169.254 starting new resolver thread.
seems like a success until it gets stuck in some resolving hostname loop
2022-10-18T18:31:48.155541Z DEBUG awscrt::task-scheduler: id=0x7f0bf0013c30: Scheduling http1_connection_outgoing_stream task for immediate execution
2022-10-18T18:31:48.155573Z DEBUG awscrt::task-scheduler: id=0x7f0bf0013c30: Running http1_connection_outgoing_stream task with <Running> status
2022-10-18T18:31:48.157535Z DEBUG awscrt::dns: static: resolved record: 52.217.203.162
2022-10-18T18:31:48.157564Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:31:48.157597Z DEBUG awscrt::dns: static: new address resolved 52.217.203.162 for host goofys-s3-oss.s3.us-east-1.amazonaws.com caching
2022-10-18T18:31:48.157631Z DEBUG awscrt::dns: static, invoking resolution callback for host goofys-s3-oss.s3.us-east-1.amazonaws.com with 1 addresses
2022-10-18T18:31:49.154776Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:31:49.154812Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:31:49.154828Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:31:49.157808Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:31:49.158365Z DEBUG awscrt::dns: static: resolved record: 52.217.203.162
2022-10-18T18:31:49.158388Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:31:50.155013Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:31:50.155009Z DEBUG awscrt::task-scheduler: id=0x7f0bf0001400: Running (null) task with <Running> status
2022-10-18T18:31:50.155061Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:31:50.155101Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:31:50.158544Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:31:50.159877Z DEBUG awscrt::dns: static: resolved record: 3.5.1.174
2022-10-18T18:31:50.159895Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:31:50.159912Z DEBUG awscrt::dns: static: new address resolved 3.5.1.174 for host goofys-s3-oss.s3.us-east-1.amazonaws.com caching
2022-10-18T18:31:51.155277Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:31:51.155315Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:31:51.155331Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:31:51.160041Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:31:51.161772Z DEBUG awscrt::dns: static: resolved record: 52.216.41.154
2022-10-18T18:31:51.161800Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:31:51.161815Z DEBUG awscrt::dns: static: new address resolved 52.216.41.154 for host goofys-s3-oss.s3.us-east-1.amazonaws.com caching
from mountpoint-s3.
jamesbornholt
commented on May 29, 2024
Can you compare that output to what you get running on just the EC2 instance to see where they're diverging?
from mountpoint-s3.
monthonk
commented on May 29, 2024
There are too many logs! but IMDS client got response 200 back when running on just EC2 instance
2022-10-18T18:51:20.109970Z DEBUG awscrt::Unknown: (id=0x555cf77ddcd0) IMDS client's token is invalid and is now updating.
2022-10-18T18:51:20.109648Z INFO awscrt::S3ClientStats: id=0x555cf78a8d30 Requests-in-flight(approx/exact):1/1 Requests-preparing:1 Requests-queued:0 Requests-network(get/put/default/total):0/0/0/0 Requests-streaming-waiting:0 Requests-streaming:0 Endpoints(in-table/allocated):1/1
2022-10-18T18:51:20.110352Z DEBUG awscrt::task-scheduler: id=0x7ff870003d68: Scheduling aws_exponential_backoff_retry_task task for immediate execution
2022-10-18T18:51:20.110367Z DEBUG awscrt::task-scheduler: id=0x7ff870003d68: Running aws_exponential_backoff_retry_task task with <Running> status
2022-10-18T18:51:20.110376Z DEBUG awscrt::exp-backoff-strategy: id=0x555cf77c8220: Vending retry_token 0x7ff870003d10
2022-10-18T18:51:20.110384Z DEBUG awscrt::Unknown: id=0x555cf77ddcd0: IMDS Client successfully acquired retry token.
...
2022-10-18T18:51:20.112163Z DEBUG awscrt::Unknown: (id=0x555cf77ddcd0) IMDS client query received http status code 200 for requester 0x7ff8700033e0.
2022-10-18T18:51:20.112188Z DEBUG awscrt::http-stream: id=0x7ff884012ad0: Client request complete, response status: 200 (OK).
from mountpoint-s3.
monthonk
commented on May 29, 2024
also resolving hostname loop might not be relevant since I'm also seeing it on a success mount.
from mountpoint-s3.
monthonk
commented on May 29, 2024
So, the problem is region config. My bucket is in eu-west-1 and so the EC2 I'm running on, but in the container it choose us-east-1 as a region.
Running just EC2 instance
2022-10-18T18:51:20.111328Z DEBUG awscrt::connection-manager: id=0x555cf77dddd0: Received new connection (id=0x7ff8840122f0) from http layer
2022-10-18T18:51:20.111348Z DEBUG awscrt::connection-manager: id=0x555cf77dddd0: Grabbing pooled connection (0x7ff8840122f0)
2022-10-18T18:51:20.111369Z DEBUG awscrt::connection-manager: id=0x555cf77dddd0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1
2022-10-18T18:51:20.111389Z DEBUG awscrt::connection-manager: id=0x555cf77dddd0: Successfully completed connection acquisition with connection id=0x7ff8840122f0
2022-10-18T18:51:20.111419Z DEBUG awscrt::http-stream: id=0x7ff884012ad0: Created client request on connection=0x7ff8840122f0: PUT /latest/api/token HTTP/1.1
2022-10-18T18:51:20.111440Z DEBUG awscrt::task-scheduler: id=0x7ff8840123e8: Scheduling http1_connection_cross_thread_work task for immediate execution
2022-10-18T18:51:20.111468Z DEBUG awscrt::task-scheduler: id=0x7ff8840123e8: Running http1_connection_cross_thread_work task with <Running> status
2022-10-18T18:51:20.111535Z DEBUG awscrt::task-scheduler: id=0x7ff884001130: Scheduling socket_written_task task for immediate execution
2022-10-18T18:51:20.111563Z DEBUG awscrt::task-scheduler: id=0x7ff884001130: Running socket_written_task task with <Running> status
2022-10-18T18:51:20.111582Z DEBUG awscrt::task-scheduler: id=0x7ff884012380: Scheduling http1_connection_outgoing_stream task for immediate execution
2022-10-18T18:51:20.111605Z DEBUG awscrt::task-scheduler: id=0x7ff884012380: Running http1_connection_outgoing_stream task with <Running> status
2022-10-18T18:51:20.111821Z DEBUG awscrt::dns: static: resolved record: 3.5.68.164
2022-10-18T18:51:20.111848Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:51:20.111863Z DEBUG awscrt::dns: static: new address resolved 3.5.68.164 for host goofys-s3-oss.s3.eu-west-1.amazonaws.com caching
2022-10-18T18:51:20.111886Z DEBUG awscrt::dns: static, invoking resolution callback for host goofys-s3-oss.s3.eu-west-1.amazonaws.com with 1 addresses
2022-10-18T18:51:20.112163Z DEBUG awscrt::Unknown: (id=0x555cf77ddcd0) IMDS client query received http status code 200 for requester 0x7ff8700033e0.
2022-10-18T18:51:20.112188Z DEBUG awscrt::http-stream: id=0x7ff884012ad0: Client request complete, response status: 200 (OK).
2022-10-18T18:51:20.112195Z INFO awscrt::http-connection: id=0x7ff8840122f0: Shutting down connection with error code 0 (AWS_ERROR_SUCCESS).
2022-10-18T18:51:20.112212Z DEBUG awscrt::task-scheduler: id=0x7ff8840015a0: Scheduling channel_shutdown task for immediate execution
2022-10-18T18:51:20.112232Z DEBUG awscrt::connection-manager: id=0x555cf77dddd0: User releasing connection (id=0x7ff8840122f0)
2022-10-18T18:51:20.112255Z DEBUG awscrt::connection-manager: id=0x555cf77dddd0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=0, open_connection_count=1, ref_count=1
2022-10-18T18:51:20.112275Z INFO awscrt::connection-manager: id=0x555cf77dddd0: Releasing connection (id=0x7ff8840122f0)
2022-10-18T18:51:20.112297Z DEBUG awscrt::exp-backoff-strategy: id=0x555cf77c8220: Initializing retry token 0x7ff884012680
2022-10-18T18:51:20.112322Z DEBUG awscrt::task-scheduler: id=0x7ff8840126d8: Scheduling aws_exponential_backoff_retry_task task for immediate execution
2022-10-18T18:51:20.112343Z DEBUG awscrt::Unknown: (id=0x555cf77ddcd0) IMDS client updated the cached token successfully.
2022-10-18T18:51:20.112364Z DEBUG awscrt::channel: id=0x7ff8840013a0: Channel shutdown is already pending, not scheduling another.
2022-10-18T18:51:20.112390Z INFO awscrt::socket: id=0x7ff884000f90 fd=26: zero read, socket is closed
2022-10-18T18:51:20.112411Z DEBUG awscrt::channel: id=0x7ff8840013a0: Channel shutdown is already pending, not scheduling another.
2022-10-18T18:51:20.112429Z DEBUG awscrt::channel: id=0x7ff8840013a0: Channel shutdown is already pending, not scheduling another.
2022-10-18T18:51:20.112449Z DEBUG awscrt::task-scheduler: id=0x7ff8840015a0: Running channel_shutdown task with <Running> status
2022-10-18T18:51:20.112468Z DEBUG awscrt::channel: id=0x7ff8840013a0: beginning shutdown process
2022-10-18T18:51:20.112485Z DEBUG awscrt::channel: id=0x7ff8840013a0: handler 0x7ff8840121c0 shutdown in read dir completed.
2022-10-18T18:51:20.112504Z DEBUG awscrt::channel: id=0x7ff8840013a0: handler 0x7ff8840122f8 shutdown in read dir completed.
2022-10-18T18:51:20.112521Z DEBUG awscrt::task-scheduler: id=0x7ff8840013c8: Scheduling (null) task for immediate execution
2022-10-18T18:51:20.112539Z DEBUG awscrt::task-scheduler: id=0x7ff8840126d8: Running aws_exponential_backoff_retry_task task with <Running> status
2022-10-18T18:51:20.112555Z DEBUG awscrt::exp-backoff-strategy: id=0x555cf77c8220: Vending retry_token 0x7ff884012680
2022-10-18T18:51:20.112571Z DEBUG awscrt::Unknown: id=0x555cf77ddcd0: IMDS Client successfully acquired retry token.
Running in container
2022-10-18T18:52:43.023049Z DEBUG awscrt::connection-manager: id=0x55f7b64fb3b0: Grabbing pooled connection (0x7f39900122f0)
2022-10-18T18:52:43.023068Z DEBUG awscrt::connection-manager: id=0x55f7b64fb3b0: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1
2022-10-18T18:52:43.023088Z DEBUG awscrt::connection-manager: id=0x55f7b64fb3b0: Successfully completed connection acquisition with connection id=0x7f39900122f0
2022-10-18T18:52:43.023120Z DEBUG awscrt::http-stream: id=0x7f3990012ad0: Created client request on connection=0x7f39900122f0: PUT /latest/api/token HTTP/1.1
2022-10-18T18:52:43.023154Z DEBUG awscrt::task-scheduler: id=0x7f39900123e8: Scheduling http1_connection_cross_thread_work task for immediate execution
2022-10-18T18:52:43.023185Z DEBUG awscrt::task-scheduler: id=0x7f39900123e8: Running http1_connection_cross_thread_work task with <Running> status
2022-10-18T18:52:43.023255Z DEBUG awscrt::task-scheduler: id=0x7f3990001130: Scheduling socket_written_task task for immediate execution
2022-10-18T18:52:43.023288Z DEBUG awscrt::task-scheduler: id=0x7f3990001130: Running socket_written_task task with <Running> status
2022-10-18T18:52:43.023317Z DEBUG awscrt::task-scheduler: id=0x7f3990012380: Scheduling http1_connection_outgoing_stream task for immediate execution
2022-10-18T18:52:43.023343Z DEBUG awscrt::task-scheduler: id=0x7f3990012380: Running http1_connection_outgoing_stream task with <Running> status
2022-10-18T18:52:43.024851Z DEBUG awscrt::dns: static: resolved record: 52.216.52.74
2022-10-18T18:52:43.024873Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:52:43.024892Z DEBUG awscrt::dns: static: new address resolved 52.216.52.74 for host goofys-s3-oss.s3.us-east-1.amazonaws.com caching
2022-10-18T18:52:43.024914Z DEBUG awscrt::dns: static, invoking resolution callback for host goofys-s3-oss.s3.us-east-1.amazonaws.com with 1 addresses
2022-10-18T18:52:44.022491Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:52:44.022533Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:52:44.022556Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:52:44.025068Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:52:44.025748Z DEBUG awscrt::dns: static: resolved record: 52.216.52.74
2022-10-18T18:52:44.025794Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:52:45.022747Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:52:45.022787Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:52:45.022804Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:52:45.024457Z DEBUG awscrt::task-scheduler: id=0x7f39900011b0: Running (null) task with <Running> status
2022-10-18T18:52:45.025951Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:52:45.026568Z DEBUG awscrt::dns: static: resolved record: 52.216.52.74
2022-10-18T18:52:45.026615Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:52:46.022985Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:52:46.023025Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:52:46.023042Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:52:46.026766Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:52:46.028530Z DEBUG awscrt::dns: static: resolved record: 52.217.162.18
2022-10-18T18:52:46.028555Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:52:46.028577Z DEBUG awscrt::dns: static: new address resolved 52.217.162.18 for host goofys-s3-oss.s3.us-east-1.amazonaws.com caching
2022-10-18T18:52:47.023232Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T18:52:47.023274Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T18:52:47.023283Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T18:52:47.028729Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com
2022-10-18T18:52:47.030295Z DEBUG awscrt::dns: static: resolved record: 52.217.197.186
2022-10-18T18:52:47.030346Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.us-east-1.amazonaws.com successful, returned 1 addresses
2022-10-18T18:52:47.030369Z DEBUG awscrt::dns: static: new address resolved 52.217.197.186 for host goofys-s3-oss.s3.us-east-1.amazonaws.com caching
from mountpoint-s3.
monthonk
commented on May 29, 2024
Hmm, I have tried option --region=eu-west-1 but that still doesn't solve the problem.
2022-10-18T19:16:06.513525Z DEBUG awscrt::connection-manager: id=0x55d11aede390: Received new connection (id=0x7f3a8c0122f0) from http layer
2022-10-18T19:16:06.513535Z DEBUG awscrt::connection-manager: id=0x55d11aede390: Grabbing pooled connection (0x7f3a8c0122f0)
2022-10-18T19:16:06.513547Z DEBUG awscrt::connection-manager: id=0x55d11aede390: snapshot - state=1, idle_connection_count=0, pending_acquire_count=0, pending_settings_count=0, pending_connect_count=0, vended_connection_count=1, open_connection_count=1, ref_count=1
2022-10-18T19:16:06.513557Z DEBUG awscrt::connection-manager: id=0x55d11aede390: Successfully completed connection acquisition with connection id=0x7f3a8c0122f0
2022-10-18T19:16:06.513586Z DEBUG awscrt::http-stream: id=0x7f3a8c012ad0: Created client request on connection=0x7f3a8c0122f0: PUT /latest/api/token HTTP/1.1
2022-10-18T19:16:06.513597Z DEBUG awscrt::task-scheduler: id=0x7f3a8c0123e8: Scheduling http1_connection_cross_thread_work task for immediate execution
2022-10-18T19:16:06.513616Z DEBUG awscrt::task-scheduler: id=0x7f3a8c0123e8: Running http1_connection_cross_thread_work task with <Running> status
2022-10-18T19:16:06.513675Z DEBUG awscrt::task-scheduler: id=0x7f3a8c001130: Scheduling socket_written_task task for immediate execution
2022-10-18T19:16:06.513692Z DEBUG awscrt::task-scheduler: id=0x7f3a8c001130: Running socket_written_task task with <Running> status
2022-10-18T19:16:06.513702Z DEBUG awscrt::task-scheduler: id=0x7f3a8c012380: Scheduling http1_connection_outgoing_stream task for immediate execution
2022-10-18T19:16:06.513717Z DEBUG awscrt::task-scheduler: id=0x7f3a8c012380: Running http1_connection_outgoing_stream task with <Running> status
2022-10-18T19:16:07.513007Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T19:16:07.513044Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T19:16:07.513072Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T19:16:07.513257Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com
2022-10-18T19:16:07.513834Z DEBUG awscrt::dns: static: resolved record: 52.92.32.218
2022-10-18T19:16:07.513862Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com successful, returned 1 addresses
2022-10-18T19:16:08.513262Z DEBUG awscrt::task-scheduler: id=0x7f3a8c0011b0: Running (null) task with <Running> status
2022-10-18T19:16:08.513268Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T19:16:08.513361Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T19:16:08.513376Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T19:16:08.514023Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com
2022-10-18T19:16:08.514637Z DEBUG awscrt::dns: static: resolved record: 52.92.32.218
2022-10-18T19:16:08.514664Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com successful, returned 1 addresses
2022-10-18T19:16:09.513558Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T19:16:09.513603Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T19:16:09.513625Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T19:16:09.514826Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com
2022-10-18T19:16:09.516367Z DEBUG awscrt::dns: static: resolved record: 52.218.105.179
2022-10-18T19:16:09.516386Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com successful, returned 1 addresses
2022-10-18T19:16:09.516397Z DEBUG awscrt::dns: static: new address resolved 52.218.105.179 for host goofys-s3-oss.s3.eu-west-1.amazonaws.com caching
2022-10-18T19:16:10.513807Z DEBUG awscrt::dns: static: resolving host 169.254.169.254
2022-10-18T19:16:10.513846Z DEBUG awscrt::dns: static: resolved record: 169.254.169.254
2022-10-18T19:16:10.513864Z DEBUG awscrt::dns: static, resolving host 169.254.169.254 successful, returned 1 addresses
2022-10-18T19:16:10.516536Z DEBUG awscrt::dns: static: resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com
2022-10-18T19:16:10.517159Z DEBUG awscrt::dns: static: resolved record: 52.218.105.179
2022-10-18T19:16:10.517181Z DEBUG awscrt::dns: static, resolving host goofys-s3-oss.s3.eu-west-1.amazonaws.com successful, returned 1 addresses
from mountpoint-s3.
jamesbornholt
commented on May 29, 2024
Looks like this problem: https://stackoverflow.com/questions/71884350/using-imds-v2-with-token-inside-docker-on-ec2-or-ecs
Asking for an IMDSv2 token directly hangs forever from inside the container:
fuseuser@191cb718cc68:~$ curl -X PUT "http://169.254.169.254/latest/api/token"
After modifying the hop limit as that question suggests, I can mount a bucket inside the container.
However, this does suggest we're missing timeouts on credentials requests somewhere. The CLI seems to fall back to IMDSv1 when this happens, though I can't see anywhere how it decides to do that.
from mountpoint-s3.
monthonk
commented on May 29, 2024
Looking at the code, this part explicitly says that we should fall back to IMDSv1 when getting an error from token acquiring process. The problem is that we never get any response or error back even the socket timeout has been set.
Also note that ECS use different credentials provider called Container credentials, I have tested this manually and it isn't affected by this issue.
from mountpoint-s3.
monthonk
commented on May 29, 2024
For comparison, aws-rust-sdk has both CONNECT_TIMEOUT and READ_TIMEOUT in their client configuration.
The read timeout seems to be missing from CRT libraries. Do you know if we can set it somewhere?
from mountpoint-s3.
dannycjones
commented on May 29, 2024
Bug opened in CRT: awslabs/aws-c-auth#182
from mountpoint-s3.
Related Issues (20)
- Operations hang indefinitely after opening too many files HOT 2
- Ability to configure maximum retry count
- Metadata Caching in case of not existing file HOT 12
- Full throughput limit is not being used HOT 7
- Need a function to monitoring the total usage of S3 mountpoint or S3 PVC HOT 1
- Slurm(srun) unable to start mountpoint HOT 11
- Support wasabi s3 storage HOT 1
- Support ovhcloud s3 high performance object storage HOT 1
- Support for writing to Google Cloud Storage buckets HOT 10
- I think there should be a cool icon for this product HOT 1
- ERROR awscrt::AuthCredentialsProvider: static: invalid credential_source property: EcsContainer HOT 3
- Assuming role configured in AWS profile does not work outside of `aws` partition (China, US Gov Cloud, etc.) HOT 4
- Regression between 1.4.0 and 1.6.0 when overwriting files with --allow-overwrite enabled HOT 10
- Can't open & append file with path
- Clarify memory usage limiting in case of `--metadata-ttl` enabled HOT 1
- Support executable files HOT 2
- > mountpoint_s3::fuse: open failed: inode error: inode 3 (full key "synmon/test_file.txt") is not writable while being read HOT 1
- Use mount-s3 as a crate dependency HOT 1
- Debug/trace logging that includes the HTTP request headers HOT 2
- support for s3 mount on ecs variant of bottle rocket HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mountpoint-s3.