Socrata WebJob doesn't meet new SNI requirement about analytics.smgov.net HOT 17 CLOSED

Sure thing. I'll also add more notes to the README regarding that

from analytics.smgov.net.

Oh come on, don't you want to solve this totally unrelated problem instead of focusing on our actual issue 😵

Great idea, I'll try that next.

from analytics.smgov.net.

Also could be solved by #32

from analytics.smgov.net.

Another possible solution to use Python 3 in Azure WebJobs

from analytics.smgov.net.

@allejo I think I have a solution for this - I tried the approach from this SO answer on a test Web App, and it worked. So I'm going to rework the socrata job to do the same and we'll see.

from analytics.smgov.net.

Sounds good

from analytics.smgov.net.

Take a look at that commit and let me know what you think - we can always spin up a test instance/slot to deploy that branch and check it out live

from analytics.smgov.net.

Travis creates the Python virtual environment for us and is using python 2.7.x so 3.4.x isn't created at the moment. I think Travis needs to be configured to use that

from analytics.smgov.net.

Hmmm, I'm not that familiar with configuring Travis. If you'd like to take this and run go for it!

from analytics.smgov.net.

@thekaveman Care to take a look at my changes? Also, can this be tested in a test Azure instance before it's deployed? Just make sure it works as expect with 3.4

from analytics.smgov.net.

@allejo Looks OK to me, though I haven't thoroughly read up on what is different from Python 2.x --> 3.x e.g. the changes to arguments of open(...). I'm sure you got it 😄

I setup a "slot" in the azure app that we should be able to deploy to. I'll be modifying the .travis.yml to target this slot temporarily so we can test.

from analytics.smgov.net.

Ok, not sure what's going on.
My change is here ccc5d50 - Travis is running through its process almost correctly see build log

This output towards the end (starting at line 1964):

Deploying application
Deploying to Azure Web App 'csmanalytics-stage'
Already up-to-date!

says that nothing is actually sent to Azure. On the Azure side, we see .kudu/deploy.sh fail because the _site folder doesn't exist:

Command: bash .kudu/deploy.sh
Using custom deployment setting for DEPLOYMENT_SOURCE custom value is '_site'.
Handling node.js deployment.
KuduSync.NET from: 'D:\home\site\repository\_site' to: 'D:\home\site\wwwroot'
Error: Could not find a part of the path 'D:\home\site\repository\_site'.
An error has occurred during web site deployment.
Kudu Sync failed
Error: Could not find a part of the path 'D:\home\site\repository\_site'.
D:\Program Files (x86)\SiteExtensions\Kudu\57.50831.2397\bin\Scripts\starter.cmd bash .kudu/deploy.sh

from analytics.smgov.net.

Does the Azure slot have an already existing git repo with a history? If you FTP into the slot, does the repository folder D:\home\site\repository have anything inside?

from analytics.smgov.net.

More info:

I had incorrectly setup the Azure slot to connect to the GitHub repo for deployment, but we want a local git repo that Travis pushes to. That has been updated. (0)

There's an Azure setting that says which branch should be used for deployment, and it defaults to master. I updated that setting for the slot to be our webjob-refactor branch. (1)

If we look at D:\home\site\repository in the slot, we do see a git repo (but nothing else):

D:\home\site\repository>ls -alF
total 4
drwxr-xr-x 1 Unknown+User Unknown+Group 0 Sep  6 18:35 ./
drwxr-xr-x 1 Unknown+User Unknown+Group 0 Sep  6 18:35 ../
drwxr-xr-x 1 Unknown+User Unknown+Group 0 Sep  6 19:02 .git/

There is no history, and no branches other than master:

D:\home\site\repository>git status
On branch master

Initial commit

nothing to commit (create/copy files and use "git add" to track)

D:\home\site\repository>git show-branch --all
No revs to be shown.

Then I came across the code that Travis uses for Azure deployment which seems to hardcode pushing to the remote's master branch. So I undid the setting from (1) above.

After re-running that same Travis build, we get the same output on the Travis side (i.e. Already up-to-date!). The deploy isn't even making it to Azure, that seemed to be a red-herring from my previously screwed up settings in Azure (0).

from analytics.smgov.net.

See also: travis-ci/dpl#438, we may be in that situation.

from analytics.smgov.net.

Well that's certainly limiting and bothersome... Can we just manually push a built site to the test slot? The built site would need the compiled Jekyll source and the virtual python environment.

from analytics.smgov.net.

@allejo:

I finally got the compiled version of this branch deployed to our test slot in Azure. Good and bad news:

Good: the changes to aggregate look good (had to add the run.cmd file to select Python3)

Bad (not really): The socrata job is still failing, although with a completely different error message. It's unclear if we have resolved the SNI issues, because the program is failing before trying to call into sodapy:

[09/07/2016 19:02:27 > a365ba: INFO] D:\local\Temp\jobs\triggered\socrata\hjbjn5vd.vez>D:\Python34\python job.py 
[09/07/2016 19:02:28 > a365ba: ERR ]   File "job.py", line 77
[09/07/2016 19:02:28 > a365ba: ERR ]     upsert_chunks = [ data['data'][x:x+soda_batch_size] for x in range(0, len(data['data']), soda_batch_size) ]
[09/07/2016 19:02:28 > a365ba: ERR ]                 ^
[09/07/2016 19:02:28 > a365ba: ERR ] SyntaxError: invalid syntax
[09/07/2016 19:02:29 > a365ba: SYS INFO] Status changed to Failed
[09/07/2016 19:02:29 > a365ba: SYS ERR ] Job failed due to exit code 1

from analytics.smgov.net.