Advertising to centralized advertise v2 times out randomly. To repli

Centralized advertise v2 is slow about advertiseserver HOT 7 CLOSED

choksi81 commented on August 12, 2024

Centralized advertise v2 is slow

from advertiseserver.

Comments (7)

choksi81 commented on August 12, 2024

Author: leonwlaw
Correction, GET requests are not fast. Performance is highly variable, it might have to do with how much traffic the advertise server is getting at a particular time.

Also adding some more details on how to replicate this.

from advertiseserver.

choksi81 commented on August 12, 2024

Author: leonwlaw
I have discussed with Monzur about this and we propose to split the network callbacks to two components: request retrieval and request handling. Request retrieval will be parallelized so that we can read from multiple clients simultaneously; slower connections will not cause other queued up connections to time out. These retrieved requests are then put into a single Queue, which is handled by a single request handling thread. This thread will then send the proper response back to the client. Request handling should be kept to within the same thread to prevent race conditions.

from advertiseserver.

choksi81 commented on August 12, 2024

Author: justinc
Do you have timing data that shows that this separation should be expected to make an impact?

from advertiseserver.

choksi81 commented on August 12, 2024

Author: leonwlaw
The following is a test I wrote that performs advertise requests in parallel. The test server and client lookups are on the same machine. I made these changes on the server end, introducing a sleep() call inbetween requests to simulate slow connections.

I ran the tests with 20 simultaneous announces.

Test file
Output file

from advertiseserver.

choksi81 commented on August 12, 2024

Author: leonwlaw
First, we need to lower the frequency of nodetrack advertisements. This is expected to make an impact, as nodetrack is advertised every 5 seconds for every node, which is roughly ~160 requests per second. What I have done in the diff above is adapt the nodetrack advertisement to not re-advertise as often, similar to what we do with the other keys. It will wait until roughly half its TTL has expired, then re-advertise. It is currently set to 300s (750s TTL).

I have prepared a unit test that checks the number of times the nodemanager advertises over a span of time, but it does not work as planned as the advertise server is really taxed. What I propose instead is modify it to be an integration test that we can run on blackbox.

from advertiseserver.

choksi81 commented on August 12, 2024

Author: leonwlaw
I expect this issue to be resolved with r6754. Monzur and I will be doing a beta push and production push soon to verify this.

from advertiseserver.

choksi81 commented on August 12, 2024

Author: leonwlaw
I've just verified that my node on beta is indeed advertising nodetrack after approximately 5 minutes.

To test, I've modified my nmadvertise.py on my test node. I've added the line that is referred to by >>>>:

        # now that I know who to announce to, send messages to annouce my IP and 
        # port to all keys I support
        for advertisepair in advertisepairs:
          advertisekey, advertisevalue = advertisepair
          try:
>>>>        servicelogger.log(str(getruntime()) + ": Advertising: " + str(advertisekey))
            advertise_announce(advertisekey, advertisevalue, adTTL)
            # mark when we advertise
            # since some keys will be dictionaries and dictionaries

Resulting nodemanager.old file:

1381851281.55:PID-7124:4.93021619282: Advertising: {'e': 65537L, 'n': 141707587269498660269796045643175314158596605062762494808990367077588611293585238679370246694884737320154056605159687011610964378003234363538612636304544886987892641969954546403034583858473083439121699370507642046376204657159777520345084924827443302613939344152632093466745137885725679851975313074720865815539L}
1381851281.55:PID-7124:[INFO]:Started
1381851282.21:PID-7124:5.59365225438: Advertising: {'e': 22599311712094481841033180665237806588790054310631222126405381271924089573908627143292516781530652411806621379822579071415593657088637116149593337977245852950266439908269276789889378874571884748852746045643368058107460021117918657542413076791486130091963112612854591789518690856746757312472362332259277422867L, 'n': 12178066700672820207562107598028055819349361776558374610887354870455226150556699526375464863913750313427968362621410763996856543211502978012978982095721782038963923296750730921093699612004441897097001474531375768746287550135361393961995082362503104883364653410631228896653666456463100850609343988203007196015297634940347643303507210312220744678194150286966282701307645064974676316167089003178325518359863344277814551559197474590483044733574329925947570794508677779986459413166439000241765225023677767754555282196241915500996842713511830954353475439209109249856644278745081047029879999022462230957427158692886317487753201883260626152112524674984510719269715422340038620826684431748131325669940064404757120601727362881317222699393408097596981355810257955915922792648825991943804005848347665699744316223963851263851853483335699321871483966176480839293125413057603561724598227617736944260269994111610286827287926594015501020767105358832476708899657514473423153377514660641699383445065369199724043380072146246537039577390659243640710339329506620575034175016766639538091937167987100329247642670588246573895990251211721839517713790413170646177246216366029853604031421932123167115444834908424556992662935981166395451031277981021820123445253L}
1381851292.21:PID-7124:15.5977889798: Advertising: {'e': 65537L, 'n': 99429618645015796530390917803237078086271056986362768817550187974364555649175659283291594684767555639091907928967385750234739666624333916703165528072430783153422679916073846882051593851951010012323357116234307690918289192099281170643069486135891777685593645790329923237236122178265608487910338170326040020197L}
1381851302.22:PID-7124:25.6008210504: Advertising: {'e': 65537L, 'n': 147824900209915254407418082384824477540838052602689049249214555399955607115316602382753558769962006630479023709553178533244868641993886551444353606796419073003902400167329901449586625967639682339307011199349178469240721280851712197829902918125800073349875029317533237441378206571014714975286346600638549428743L}
1381851302.5:PID-7124:25.8874466531: Advertising: nodetrack
1381851583.29:PID-7124:306.668042154: Advertising: {'e': 65537L, 'n': 141707587269498660269796045643175314158596605062762494808990367077588611293585238679370246694884737320154056605159687011610964378003234363538612636304544886987892641969954546403034583858473083439121699370507642046376204657159777520345084924827443302613939344152632093466745137885725679851975313074720865815539L}
1381851598.3:PID-7124:321.68059533: Advertising: {'e': 22599311712094481841033180665237806588790054310631222126405381271924089573908627143292516781530652411806621379822579071415593657088637116149593337977245852950266439908269276789889378874571884748852746045643368058107460021117918657542413076791486130091963112612854591789518690856746757312472362332259277422867L, 'n': 12178066700672820207562107598028055819349361776558374610887354870455226150556699526375464863913750313427968362621410763996856543211502978012978982095721782038963923296750730921093699612004441897097001474531375768746287550135361393961995082362503104883364653410631228896653666456463100850609343988203007196015297634940347643303507210312220744678194150286966282701307645064974676316167089003178325518359863344277814551559197474590483044733574329925947570794508677779986459413166439000241765225023677767754555282196241915500996842713511830954353475439209109249856644278745081047029879999022462230957427158692886317487753201883260626152112524674984510719269715422340038620826684431748131325669940064404757120601727362881317222699393408097596981355810257955915922792648825991943804005848347665699744316223963851263851853483335699321871483966176480839293125413057603561724598227617736944260269994111610286827287926594015501020767105358832476708899657514473423153377514660641699383445065369199724043380072146246537039577390659243640710339329506620575034175016766639538091937167987100329247642670588246573895990251211721839517713790413170646177246216366029853604031421932123167115444834908424556992662935981166395451031277981021820123445253L}
1381851603.86:PID-7124:327.240565019: Advertising: {'e': 65537L, 'n': 99429618645015796530390917803237078086271056986362768817550187974364555649175659283291594684767555639091907928967385750234739666624333916703165528072430783153422679916073846882051593851951010012323357116234307690918289192099281170643069486135891777685593645790329923237236122178265608487910338170326040020197L}
1381851604.21:PID-7124:327.588067743: Advertising: {'e': 65537L, 'n': 147824900209915254407418082384824477540838052602689049249214555399955607115316602382753558769962006630479023709553178533244868641993886551444353606796419073003902400167329901449586625967639682339307011199349178469240721280851712197829902918125800073349875029317533237441378206571014714975286346600638549428743L}
1381851613.44:PID-7124:336.824001355: Advertising: nodetrack

from advertiseserver.

Centralized advertise v2 is slow about advertiseserver HOT 7 CLOSED

Comments (7)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent