Comments (2)
The API that routinely hits 429s is the get of Folder, Notebook, Files with the most objects. We can leave the numThreads high on most other API calls and rate limit the ones with the most Folder, Notebook, Files.
Additionally, put in a request to only move certain Folder, Notebook, and Files. For instance if you can skip over everyone's personal notebooks that will greatly request the number of API requests you'll need to make.
from ucx.
I've added the relevant retries and rate limiting logic to the recent version.
from ucx.
Related Issues (20)
- [FEATURE]: Create EXTERNAL LOCATIONs to map to External Tables (AWS)
- [BUG]: Cannot Install/Uninstall: "Cluster policy not present"/ "Missing required field: policy_id"
- [BUG]: External Metastore not detected if it is only configured in Global SQL Settings
- [BUG]: Attached wheel in UCX Policy breaks assessment job
- [CUJ]: Milestones for table migration - Azure
- [FEATURE]: Command to re-map cluster type HOT 1
- [BUG]: AttributeError: type object ‘WorkspaceConfig’ has no attribute ‘from_file’ HOT 2
- [FEATURE]: List all jobs, not only the incompatible ones HOT 1
- [FEATURE]: Have a filter for schema names
- [FEATURE]: Write `upgraded_to` properties to HMS table may not work due to permission issue HOT 2
- [BUG]: Groups in config file not considered for group migration job HOT 1
- [BUG]: During installation, ucx does not open config file in the browser even when prompted.
- [BUG]: Troubleshooting code in `docs/local-group-migration.md` needs updating
- [BUG]: `file`, `abfss`, `gcs`, and `s3` init scripts are not scanned
- [BUG]: ERROR installing components task failed: 'policy_id' must be supplied
- [BUG]: Wheel file is not added as a dependent library in job tasks
- [BUG]: tacl cluster fails to install the library HOT 6
- [FEATURE]: Add `RemoveAfter` tag on all UCX jobs, so that they're cleaned up by `watchdog` even if they are in the approved instance pool
- [BUG]: Alias in projecting fails to be resolved
- [BUG]: Upgrading UCX to 13.2 via CLI fails HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ucx.