hey. is it possible to get blobxfer that when uploading a directory from my local machine, to delete remote files which are no longer local present? this way i could use blobxfer for backup purposes and make sure the remote tree is identical to my local one

it will prompt the "there is unsupport parameter" error when throw out the create_pool rest call.

Receiving "AADSTS90093: User cannot consent to web app requesting user impersonation as an app permission." Is Administrative access required for AAD tenant to run this project?

It seems 'User subscription' batch have the only way to connect through AAD https://blogs.technet.microsoft.com/windowshpc/2017/03/17/azure-batch-vnet-and-custom-image-support-for-virtual-machine-pools/

How can I do it with Batch Explorer?

Currently one cannot specify any details of the user account used to execute the coordination command. It would be very useful to allow this to be set to an administrator account for configuring system services (e.g. a shared filesystem for the task)

Current work around is adding a script and making it setuid root with a pool start task, but that's horrible.

Hi,

I have task with 100 jobs and I want to download all the stdout.txt files.

I tried to do it using this python command :

txtfile = urllib2.urlopen('https://barilanbatchhpcuseast.eastus.batch.azure.com/jobs/Job1/tasks/task1_0_1/files/stdout.txt').read()

But i got the following error : urllib2.HTTPError: HTTP Error 400: A query parameter that's mandatory for this request is not specified

What can be the problem?

create container and skip on MD5 match are both True by default.
So, when running a command like:

blobxfer storage_account new_container /path/to/dir --upload --storageaccountkey=${AZURE_STORAGE_ACCOUNT_KEY} 

the upload fails because the script, first tries to retrieve the blob listings from the container. But it will raise an azure.common.AzureMissingResourceHttpError and stop the execution.

Please can you provide a Batch Explorer tool for our Operations team to use.

Thanks

Hi,

i set the proxy for the blob_client (python) with

blob_client.set_proxy("REDACTED",8080)

And for the storage everything is working fine. But for the batch_client in the creation of the pool i just wait and after some time i get a timeout exception from the urllib3 library.

Hello Guys

When the new person is downloading the batch explorer code and trying to run first time they are getting exceptions. because the code is trying to de-serialize the xml node and since file is empty created with zero bytes , the code was exploding.

I manually created the blank account xml file in the app data directory. C:\Users\AppData\Local\Microsoft\Azure Batch Explorer with following code. then ran again and then the app worked.

`

-

`

So please fix the code here to not only check if file exists and if it is not zero bytes then de-serialize otherwise write the xml lines first and then de-serialize.

private async Task LoadAccountsAsync()
{
string accountFileName = GetAccountFileName();
if (!File.Exists(accountFileName) OR ALSO CHECK IF FILE EXISTS AS ZERO BYTES )
{
using (var myFile = File.Create(accountFileName))
{
}
await this.SaveAccountsAsync().ConfigureAwait(false);
}

        //Try first to deserialize using the new account deserialization mechanism, if that fails, fall back to the old way
        bool writeBackToFile = false;
        try
        {
            XmlSerializer x = new XmlSerializer(typeof(DefaultAccountSerializationContainer));
            using (StreamReader reader = new StreamReader(accountFileName))
            {
                object deserializedObject = x.Deserialize(reader); //TODO: This is not technically async...

                DefaultAccountSerializationContainer accountContainer = (DefaultAccountSerializationContainer)deserializedObject;

                List<Account> 

Hi guys,

I'm trying to use Azure Batch to run a Job once a day. The input will be three files with different sets of data, that I would like to process in parallel.

To achieve this I'm trying to use JobSchedules, but I don't know how to create one Task for each file using a JobManagerTask. I didn't find any documentation about JobSchedule that covers this either.

Do you have some examples that I can use ?

Thanks!

If you add a resource file to the coordination command for a multi-instance task then this is not downloaded to the task working directory. Instead, it is added to the parent of this (where the task stderr.txt etc are placed). This wouldn’t be a problem except that the working directory for coordination command execution is the task working directory. (Checked by setting my command string to be pwd && sh ../coordination.sh).

As a work-around you can just cd .. first, but it's quite annoying.

This is for the agent SKU ID "batch.node.centos 7", in case this is specific to that.

After downloading the latest fix for the issue that was recently closed with #82, I ran into another issue:

HTTPSConnectionPool(host='sampleblobstore.blob.core.windows.net', port=443): 
Max retries exceeded with url: /xxxxxxxxxxxxuyaidlajdsflaksfcc0dc9fe4-671141903_p.jpg?blockid=MDAwMDAwMDA=&comp=block 
(Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x44834850>: 
Failed to establish a new connection: [Errno -2] Name or service not known',))

This is a largish folder that I've been trying to upload to Azure. All the other smaller folders were uploaded fine.

My command I used:

blobxfer.py --storageaccountkey $AZURE_STORAGE_ACCESS_KEY $AZURE_STORAGE_ACCOUNT $AZURE_CONTAINER_NAME .

The storage account key is taken in as a command line parameter only for blobxfer. Which implies that the storageaccountkey which is a secret for the account is visible in the ps aux output, for big uploads. This a security issue, other clients(e.g. the aws client for s3) get around this issue by accepting the secrets as a environment variable or a config file.

Thoughts?

cc @Dagrol

The Python article tutorial currently monitors tasks for completion - however does not monitor whether the nodes on the pool successfully start up - if start task fails then the tasks will never start leaving the script to poll for 20 minutes then exit without helpful feedback to the user.

i'm just testing uploading using blobxfer. I have no issue downloading data, but can't get the upload to work. I'm sure the SAS has write permission on the container, and I've also just used the storage account key to no effect. Here's what I'm running (foo.txt isn't empty).

$ blobxfer account container foo.txt --no-overwrite --numworkers 1 --saskey $BLOB_SAS_KEY --upload --no-createcontainer --no-computefilemd5
=======================================
 azure blobxfer parameters [v0.10.0]
=======================================
             platform: Darwin-15.4.0-x86_64-i386-64bit
   python interpreter: CPython 2.7.11
     package versions: az.common=1.1.1 az.sml=0.20.2 az.stor=0.30.0 crypt=1.3.1 req=2.9.1
      subscription id: None
      management cert: None
   transfer direction: local->Azure
       local resource: foo.txt
      include pattern: None
      remote resource: None
   max num of workers: 1
              timeout: None
      storage account: account
              use SAS: True
  upload as page blob: False
  auto vhd->page blob: False
            container: container
   blob container URI: https://account.blob.core.windows.net/container
     compute file MD5: False
    skip on MD5 match: True
   chunk size (bytes): 4194304
     create container: False
  keep mismatched MD5: False
     recursive if dir: True
component strip on up: 1
        remote delete: False
           collate to: disabled
      local overwrite: False
      encryption mode: disabled
         RSA key file: disabled
         RSA key type: disabled
=======================================

script start time: 2016-05-05 18:18:45

If I kill the job, it looks to be hung-up making the ssl connection:
script start time: 2016-05-05 18:18:45
^CTraceback (most recent call last):
File "/Users/clackner/anaconda/envs/poc/bin/blobxfer", line 11, in
sys.exit(main())
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 2118, in main
blobskipdict = get_blob_listing(blob_service[0], args)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 1560, in get_blob_listing
container_name=args.container, marker=marker, include=incl)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 1373, in azure_request
return req(_args, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 538, in list_blobs
requests.get, url=url, params=reqparams, timeout=self.timeout)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/blobxfer.py", line 1373, in azure_request
return req(_args, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/api.py", line 67, in get
return request('get', url, params=params, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, *_send_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/httplib.py", line 1136, in getresponse
response.begin()
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 180, in recv
data = self.connection.recv(_args, *_kwargs)
File "/Users/clackner/anaconda/envs/poc/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1319, in recv
result = _lib.SSL_read(self._ssl, buf, bufsiz)
KeyboardInterrupt

Node shutdown through the API is inconsistent. Some instances have failed to shutdown for up to 10 times, then succeeded at shutting down. The classic and the new portal GUI's don't help either.

+ Getting virtual machines                                                     
+ Shutting down VM                                                             
info:    vm shutdown command OK
info:    Executing command vm shutdown
+ Getting virtual machines                                                     
+ Shutting down VM                                                             
error:   InternalError : The server encountered an internal error. Please retry the request.
info:    Error information has been recorded to /Users/buraky/.azure/azure.err
error:   vm shutdown command failed

Hi,

Maybe I'm wrong but I think that makes no difference when the option --keeprootdir is enabled or not.

I am using a modified version of blobxfer, in my version I can select with a number the start point of the local path in order to create the virtual path on azure.

Example:

/home/user/azure/ folder1
folder2/subfolder1

With my version, selecting 3 as start point, I can create the following virtual path on azure:

https://xxx.blob.core.windows.net/container/folder1
https://xxx.blob.core.windows.net/container/folder2/subfolder1

Instead of:

https://xxx.blob.core.windows.net/container/home/user/azure/folder1
https://xxx.blob.core.windows.net/container/home/user/azure/folder2/subfolder1

I can pull a request and add this option to the flag --keeprootdir like:
--keeprootdir [STARTPOINT] (with 1 as default)

Regards,

As per this guide, application litware was created as demo/example, can you please release its source code for developers to learn and debug?

I just tried to run the python sample with azure 2.0.0rc5.
I get the following error

me@camnetwork:/tmp/azure-batch-samples/Python/Batch/article_samples$ python python_tutorial_client.py
Sample start: 2016-07-29 12:07:48

Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/python_tutorial_task.py to container [application]...
Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/data/taskdata1.txt to container [input]...
Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/data/taskdata2.txt to container [input]...
Uploading file /tmp/azure-batch-samples/Python/Batch/article_samples/data/taskdata3.txt to container [input]...
Traceback (most recent call last):
File "python_tutorial_client.py", line 480, in
base_url=_BATCH_ACCOUNT_URL)
File "/tmp/sandbox/local/lib/python2.7/site-packages/azure/batch/batch_service_client.py", line 137, in init
self._serialize = Serializer(client_models)
TypeError: init() takes exactly 1 argument (2 given)

Getting TypeError on the xrange command. numworkers appears to be set to 64 (default), maybe a problem with assigning maxworkers here?

maxworkers = min([args.numworkers, nstorageops])

Error below...

=======================================
      azure blob xfer parameters
=======================================
     subscription id: None
     management cert: None
  transfer direction: local->Azure
      local resource: libfoo.deb
     remote resource: None
  max num of workers: 64
             timeout: None
     storage account: foo
             use SAS: True
 upload as page blob: False
 auto vhd->page blob: False
           container: packages
  blob container URI: https://foo.blob.core.windows.net/packages
    compute file MD5: True
   skip on MD5 match: True
  chunk size (bytes): 4194304
    create container: False
 keep mismatched MD5: False
    recursive if dir: True
 keep root dir on up: False
=======================================

script start time: 2015-09-04 19:50:17
foo.deb md5: ThWXPkbFTO1HAeddfZUv8A==
foo-dev_.deb->foo-dev.deb: local ThWXPkbFTO1HAeddfZUv8A== remote ThWXPkbFTO1HAeddfZUv8A== ->match, skipping upload
performing None put blocks and 0 put block lists
Traceback (most recent call last):
  File "blobxfer.py", line 1406, in <module>
    main()
  File "blobxfer.py", line 1205, in main
    for _ in xrange(maxworkers):
TypeError: an integer is required
Failure occurred communicating with 

Hi,
the problems from the closed issue 131 seems to persist. Both using the BatchExplorer and the usual batch rest API, the tasks have statistics, but the jobs don't.

Receiving error when transferring blob:

Traceback (most recent call last):
File "blobxfer.py", line 1310, in
main()
File "blobxfer.py", line 980, in main
blobskipdict = get_blob_listing(blob_service, args)
File "blobxfer.py", line 664, in get_blob_listing
maxresults=_MAX_LISTBLOBS_RESULTS)
File "blobxfer.py", line 547, in azure_request
raise exc
AttributeError: 'module' object has no attribute '_parse_blob_enum_results_list'

This problem starting coincides with the latest release of the Azure SDK for Python (1.0) and the problem goes away when rolling back to 0.11.1 version.

Look at:
https://github.com/Azure/azure-storage-python/releases

Interesting that the namespace has changed from azure.storage to azure.storage.blob -- could just be that?

Thanks,
John

I'm trying to run DoNetTutorial on my azure account. I have subscription.

It is giving this error while task being started.
The pool was created.
The job was created.
The 3 nodes were created.

But they failed.

Can you help me ?

Stripping local directory structure or set Azure blob storage destination directory on upload as AzCopy
AzCopy /Source: /Dest: /Pattern:

The samples and accompanying tutorial at https://docs.microsoft.com/en-us/azure/batch/batch-python-tutorial need to be updated for v3.0.0 of the Azure Batch Client for Python. For example, target_dedicated and current_dedicated on CloudPool, AddPoolParameter and PoolSpecification have been renamed to target_dedicated_nodes and current_dedicated_nodes.

i have some blob name like "https://xxx.blob.core.windows.net/my-container/Top%2024%20P/BV.json"

when i use blobxfer it error like this

remote file TOP 20 P/ length: 0 bytes, md5: 1B2M2Y8AsgTpgAmY7PhCfg==
Traceback (most recent call last):
  File "/usr/local/bin/blobxfer", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/blobxfer.py", line 1245, in main
    blob, blobdict[blob][0], blobdict[blob][1], False)
  File "/usr/local/lib/python3.4/dist-packages/blobxfer.py", line 840, in generate_xferspec_download
    lmd5 = compute_md5_for_file_asbase64(localfile)
  File "/usr/local/lib/python3.4/dist-packages/blobxfer.py", line 711, in compute_md5_for_file_asbase64
    with open(filename, 'rb') as filedesc:
IsADirectoryError: [Errno 21] Is a directory: '/var/www/html/my-container/TOP 20 P/'

Is there a way to get the __version__ from the Batch Python client lib? e.g. 2.0.1? If not, it would be helpful to implement __version__ in the Batch Python client lib so calling code can detect and work around SDK changes.

It would be awesome to have these updated for the 4.0 SDK.

Thanks for all your work/help!

i have some zero byte file in my blob.
when i try to download my blob by 'blobxfer.py', it show error as below.

remote file graph/FER-F.txt length: 0 bytes, md5:
Traceback (most recent call last):
  File "blobxfer.py", line 1404, in <module>
    main()
  File "blobxfer.py", line 1173, in main
    blob, blobdict[blob][0], blobdict[blob][1], False)
  File "blobxfer.py", line 820, in generate_xferspec_download
    filedesc.seek(contentlength - 1)
IOError: [Errno 22] Invalid argument

The C# MPI Article sample includes a .vcxproj example MSMPI application. This project is currently ignored by CI. We should investigate how to best handle it. MSBuild does support the .vcxproj format, but we need to see how easy it is to integrate with our AppVeyor CI build.

Getting the following error with C# Sample in jobs.cs

CloudStorageAccount cloudStorageAccount = new CloudStorageAccount(
new StorageCredentials(
accountSettings.StorageAccountName,
accountSettings.StorageAccountKey),
accountSettings.StorageServiceUrl,
useHttps: true);

System.ArgumentException: 'Primary and secondary location URIs in a StorageUri must point to the same resource.
Parameter name: secondaryUri'

There seems to be an issue when using the blobep switch. Running like this

blobxfer.py --storageaccountkey test storage file.mp4 --upload --blobep blob.core.chinacloudapi.cn

Throws this error

Traceback (most recent call last):
File "./blobxfer.py", line 1310, in
main()
File "./blobxfer.py", line 980, in main
blobskipdict = get_blob_listing(blob_service, args)
File "./blobxfer.py", line 664, in get_blob_listing
maxresults=_MAX_LISTBLOBS_RESULTS)
File "./blobxfer.py", line 525, in azure_request
return req(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/blobservice.py", line 485, in list_blobs
response = self._perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 143, in _perform_request
resp = self._filter(request)
File "/usr/local/lib/python2.7/dist-packages/azure/storage/storageclient.py", line 132, in _perform_request_worker
return self._httpclient.perform_request(request)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 192, in perform_request
self.send_request_headers(connection, request.headers)
File "/usr/local/lib/python2.7/dist-packages/azure/http/httpclient.py", line 171, in send_request_headers
connection.endheaders()
File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 814, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 776, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 1157, in connect
self.timeout, self.source_address)
File "/usr/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno -2] Name or service not known

Even tho I can resolve the storage URL on the same server that this is running on. Any help would be great.

I am trying to run python_tutorial_client.py and it says:
ModuleNotFoundError: No module named 'common.helpers'
I guess the problem is in this line:
import common.helpers # noqa
I tried installing the common module with pip install common, which didn't help although the library common got installed.
It was confusing to find out that there is this library named common, but there's also azure.common from the Azure SDK.
Anyways, the import in the python file did not start working even after installing azure.common and changing the non-working line to import azure.common.helpers or from azure.common import helpers (the last one gives ImportError: cannot import name 'helpers').
I am using Python 3.6.2 on Windows 10.

As I was using SAS to upload a 1Gb tsv file to a container, it fail. Is that any problem to do the upload or i am missing something need to do?

There is no statistics for jobs. They are visible for individual tasks though.

Could you please provide more information on how to prepare these files?

• wget http://.blob.core.windows.net//OpenFOAM_CentOS7_HPC.tgz
• wget http://.blob.core.windows.net//OpenFOAM_CentOS7_HPC_libs.tgz
• wget http://.blob.core.windows.net//motorBike_3M.tgz
• wget http://.blob.core.windows.net//ParaView-3.14.1-Linux-64bit.tar.gz

What they actually contain and How did you generate these files?

Is Intel MPI a part of CentOS images?

Thanks,
lumirosl @ MSFT

When using the latest version of Microsoft.WindowsAzure.Storage (7.1.2.0) and Microsoft.Azure.Batch (5.0.0.0), this solution does not compile.

Issue #1 - "WhenAllAsync" does not exist in the new Batch DLL:

bool timedOut = await taskStateMonitor.WhenAllAsync(tasks, TaskState.Completed, timeout);

Issue #2 - CloudBlockBlob does not have a method that matches this signature:
await blobData.UploadFromFileAsync(filePath, FileMode.Open);

I can't download a file from task node. I'am getting this exeception:

Microsoft.Azure.Batch.Common.BatchException: Operation returned an invalid status code 'Forbidden' ---> Microsoft.Azure.Batch.Protocol.Models.BatchErrorException: Operation returned an invalid status code 'Forbidden'
at Microsoft.Azure.Batch.Protocol.FileOperations.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Batch.Protocol.BatchRequestBase2.<ExecuteRequestWithCancellationAsync>d__42.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Batch.Protocol.BatchRequestBase2.d__40.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.Azure.Batch.Protocol.BatchRequestBase2.<ExecuteRequestAsync>d__40.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Batch.ProtocolLayer.<ProcessAndExecuteBatchRequest>d__841.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Batch.ProtocolLayer.d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.Batch.JobOperations.d__39.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.BatchExplorer.Models.TaskModel.d__58.MoveNext() in C:\Users\g.pedral.santos\Desktop\azure batch\azure-batch-samples-master\CSharp\BatchExplorer\Models\TaskModel.cs:line 384
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.BatchExplorer.Models.TaskModel.d__56.MoveNext() in C:\Users\g.pedral.santos\Desktop\azure batch\azure-batch-samples-master\CSharp\BatchExplorer\Models\TaskModel.cs:line 356
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.BatchExplorer.ViewModels.MainViewModel.d__215.MoveNext() in C:\Users\g.pedral.santos\Desktop\azure batch\azure-batch-samples-master\CSharp\BatchExplorer\ViewModels\MainViewModel.cs:line 1728
Request Information
ClientRequestId:
RequestId:6cbffd54-1fc7-48cd-a56f-4ec3a5ab9cae
HttpStatusCode:Forbidden
StatusMessage:Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

How do I set blob properties and metadata at file level ? and also if I want to set it for all the files under a folder ?

This check:
if args.subscriptionid is not None and args.managementcert is None: raise ValueError( 'cannot specify subscription id without a management cert')

is unnecessary for the following usage:
python $blobxfer_location/blobxfer.py --timeout 1800 --numworkers $NUM_WORKERS --subscriptionid $subscriptionid --storageaccountkey $storage_account --download --remoteresource

It seems that Batch Explorer has a bug that when trying to delete a pool nothing happens. A colleague confirmed that. Deleting the pool via Azure Portal works, and also in my own Python code using the azure-batch library.

Getting strange error when uploading

.7/dist-packages/requests/packages/urllib3/util/ssl_.py:100: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
xfer progress: [>>>>>>>>>>>>                  ] 40.00%    3275.82 blocks/min    The value for one of the HTTP headers is not in the correct format.
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:9ab12df3-0001-0036-7f50-200b01000000
Time:2015-11-16T09:21:10.3991192Z</Message><HeaderName>x-ms-blob-content-length</HeaderName><HeaderValue>0</HeaderValue></Error>

Time:2015-11-17T07:44:37.7457230Z
The specified blob or block content is invalid.

RequestId:445ea4c3-0001-00ed-330b-217d34000000
Time:2015-11-17T07:44:32.1473355Z

Using the python blobxfer, many times and with many large files (but always smaller than 190G), I receive the error "The request body is too large and exceeds the maximum permissible limit"
I can upload some large files, but is not possible for others files, doesn't matter how many retries or the number of workers.

Also, I have this problem when the local resource to upload is a folder (a folder containing one single file), but if I try to upload the same file specifying the file, the error doesn't appear.

I am using the following python packages:

pip freeze
azure==1.0.2
azure-common==1.0.0
azure-mgmt==0.20.1
azure-mgmt-common==0.20.0
azure-mgmt-compute==0.20.0
azure-mgmt-network==0.20.1
azure-mgmt-nspkg==1.0.0
azure-mgmt-resource==0.20.1
azure-mgmt-storage==0.20.0
azure-nspkg==1.0.0
azure-servicebus==0.20.1
azure-servicemanagement-legacy==0.20.1
azure-storage==0.20.2
elasticsearch==2.2.0
futures==3.0.3
python-dateutil==2.4.2
requests==2.9.1
six==1.10.0
urllib3==1.14
wheel==0.26.0

And blobxfer.py v0.9.9.5

As an example:

azure blobxfer parameters [v0.9.9.5]
subscription id: None
management cert: None
transfer direction: local->Azure
local resource: archive/
remote resource: None
max num of workers: 4
timeout: None
storage account: ---
use SAS: False
upload as page blob: False
auto vhd->page blob: False
container: ---
blob container URI: https://---.blob.core.windows.net/---
compute file MD5: True
skip on MD5 match: True
chunk size (bytes): 4194304
create container: True
keep mismatched MD5: False
recursive if dir: True
keep root dir on up: False
collate to: disabled

script start time: 2016-01-29 10:00:09

g--.tar.gz md5: lhD55kDeLW9uh4PXtJ7LhQ==
detected 0 empty files to upload
performing 25600 put blocks/blobs and 1 put block lists
xfer progress: [ ] 0.00% 0.00 blocks/min The request body is too large and exceeds the maximum permissible limit.

RequestId:a6af7e74-0001-00f6-0474-5ae0d5000000
Time:2016-01-29T09:06:25.4964043Z100000

ls -lah
-rw-r--r-- 1 root root 100G Jan 26 15:46 g--.tar.gz

If the local file has a question mark in its name the upload can stack trace:

$ blobxfer --version
0.9.9.8

$ mkdir x

$ echo x > 'x/xxx?.xxx'

$ blobxfer --storageaccountkey 'redacted' redacted container x
Traceback (most recent call last):
File "/redacted/local/lib/python2.7/site-packages/blobxfer.py", line 877, in run
offset, bytestoxfer, encparam, flock, filedesc)
File "/redacted/local/lib/python2.7/site-packages/blobxfer.py", line 1002, in putblobdata
content_md5=contentmd5, timeout=self.timeout)
File "/redacted/local/lib/python2.7/site-packages/blobxfer.py", line 1309, in azure_request
return req(_args, *_kwargs)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/blob/blobservice.py", line 2366, in put_block
self._perform_request(request)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/storageclient.py", line 178, in _perform_request
_storage_error_handler(ex)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/_serialization.py", line 25, in _storage_error_handler
return _general_error_handler(http_error)
File "/redacted/local/lib/python2.7/site-packages/azure/storage/_common_error.py", line 82, in _general_error_handler
raise AzureHttpError(message, http_error.status)
AzureHttpError: An HTTP header that's mandatory for this request is not specified.

RequestId:redacted
Time:2016-01-24T17:47:06.9200167Zx-ms-blob-type

Removing the '?' makes the upload succeed.

First and foremost, I really like blobxfer. I've only just discovered it but it really hits a sweet spot for a handful of use cases I have right now. I came across a small issue in the README (https://github.com/Azure/azure-batch-samples/blob/master/Python/Storage/README.rst ).

Should:
blobxfer mystorageacct container0 mylocaldir --remoteresource .

Be:
blobxfer mystorageacct container0 mylocaldir --download --remoteresource .

The former wrote various temp files to be container and it ended up being a bit confusing.

As per the doc, it will wait until its members has reached a desired state at least once.

Here comes my code (I know its dirty) ---

var tasksDetail = new ODATADetailLevel(selectClause: "id,executionInfo");
var tasks = await batchClient.JobOperations.ListTasks(jobId, tasksDetail).ToListAsync();
Parallel.ForEach(tasks, (task) =>
{
    if (task == null)
    {
        Logger.Warn("Invalid Task info");
        return;
    }

    if (task.State == TaskState.Completed)
    {
        Logger.Info($"Task '{task.Id}' is already completed, skipping monitoring");
        return;
    }

    // Wait for 'Running' state
    batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Running, timeout).Wait();
    task.RefreshAsync(new ODATADetailLevel(selectClause: "id,nodeInfo")).Wait();
    Logger.Debug($"Task '{task.Id}' is running on Node '{task.ComputeNodeInformation?.ComputeNodeId}' now");

    // Wait for 'Completed' state
    batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Completed, timeout).Wait();
    task.RefreshAsync(new ODATADetailLevel(selectClause: "id,executionInfo")).Wait();
    Logger.Info($"Task '{task.Id}' is completed now, duration: {task.ExecutionInformation.EndTime - task.ExecutionInformation.StartTime:c}");
});

This code snippet works well. But when I tried to switch 'Running' and 'Completed' section (like below), it blocked while waiting for 'Running' state. IMHO, this behavior differs from what the document describes ---
the task first became 'Running' and then 'Completed', so after it reached 'Completed' state, it must have already reached 'Running' and should not be blocked at batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Running, timeout).Wait();.

    // Wait for 'Completed' state
    batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Completed, timeout).Wait();
    task.RefreshAsync(new ODATADetailLevel(selectClause: "id,executionInfo")).Wait();
    Logger.Info($"Task '{task.Id}' is completed now, duration: {task.ExecutionInformation.EndTime - task.ExecutionInformation.StartTime:c}");

    // Wait for 'Running' state
    batchClient.Utilities.CreateTaskStateMonitor().WhenAll(new[] {task}, TaskState.Running, timeout).Wait();
    task.RefreshAsync(new ODATADetailLevel(selectClause: "id,nodeInfo")).Wait();
    Logger.Debug($"Task '{task.Id}' is running on Node '{task.ComputeNodeInformation?.ComputeNodeId}' now");

I have gotten all of the other samples to work with the exception of Python Azure Batch Sample 4. The pool gets created successfully, it is the key generation step that keeps failing with file not found. I replaced ssh-keygen with ls command only and it still failed. I think I have the latest version, maybe not?

Figured it out, please disregard,

1. Accounts, Add, Default Account Manager
2. Fill in fields with a,b,c,d.
3. Click OK button, note window does not dismiss
4. Retry steps with Debug binaries, note feature works as designed