Comments (5)
Definitely, we should keep @aMahanna's example! It's already available on this issue, which is going to be indexed by search engines, since the project is public. People will be able to get here through a search.
The HTTP Clients page already contains a generic example. We should keep it as simple as possible, therefore I would be hesitant to add the UnixSocketsHTTPClient
on top of that.
Nevertheless, you are more than welcome to share your expertise with our community. I'll be moving this thread into a "Show and tell" discussion here.
Please feel free share more info, gists or any relevant code snippets. Or, if you have a blog post where you wrote in more detail about Unix sockets in the context of ArangoDB, please do share it. All contributions are much appreciated!
from python-arango.
Hey @sausix!
This should be possible given that the ArangoClient constructor supports user-defined HTTP Clients, so we might be able to get away with creating a UnixSocketsHTTPClient
that overrides the create_session()
and send_request()
methods to handle a Unix connection.
Although not natively supported in requests
, you could consider the requests-unixsocket package as a workaround.
Here is what the DefaultHTTPClient in python-arango
looks like, and here is what I can suggest as a draft for the UnixSocketsHTTPClient
:
from typing import MutableMapping, Optional, Tuple, Union
from urllib.parse import urlparse
import requests_unixsocket
from requests import Session
from requests_toolbelt import MultipartEncoder
from arango.http import DefaultHTTPClient
from arango.response import Response
from arango.typings import Headers
DEFAULT_REQUEST_TIMEOUT = 60
class UnixSocketsHTTPClient(DefaultHTTPClient):
"""Unix Socket HTTP client implementation.
:param unix_socket_url: Unix socket URL.
:type unix_socket_url: str
:param request_timeout: Timeout in seconds for each individual connection.
:type request_timeout: int | float
:param retry_attempts: Number of retry attempts.
:type retry_attempts: int
:param backoff_factor: Backoff factor for retry attempts.
:type backoff_factor: float
:param pool_connections: The number of urllib3 connection pools to cache.
:type pool_connections: int
:param pool_maxsize: The maximum number of connections to save in the pool.
:type pool_maxsize: int
:param pool_timeout: If set, then the pool will be set to block=True,
and requests will block for pool_timeout seconds and raise
EmptyPoolError if no connection is available within the time period.
:type pool_timeout: int | float | None
"""
def __init__(
self,
unix_socket_url: str,
request_timeout: Union[int, float, None] = DEFAULT_REQUEST_TIMEOUT,
retry_attempts: int = 3,
backoff_factor: float = 1.0,
pool_connections: int = 10,
pool_maxsize: int = 10,
pool_timeout: Union[int, float, None] = None,
) -> None:
self.unix_socket_url = unix_socket_url
self.request_timeout = request_timeout
self._retry_attempts = retry_attempts
self._backoff_factor = backoff_factor
self._pool_connections = pool_connections
self._pool_maxsize = pool_maxsize
self._pool_timeout = pool_timeout
def create_session(self, host: str) -> Session:
"""Create and return a new session/connection.
:param host: ArangoDB host URL.
:type host: str
:returns: requests session object
:rtype: requests.Session
"""
# retry_strategy = Retry(
# total=self._retry_attempts,
# backoff_factor=self._backoff_factor,
# status_forcelist=[429, 500, 502, 503, 504],
# allowed_methods=["HEAD", "GET", "OPTIONS"],
# )
unix_adapter = requests_unixsocket.UnixAdapter(
self.request_timeout,
self._pool_connections,
# connection_timeout=self.request_timeout,
# pool_maxsize=self._pool_maxsize,
# pool_timeout=self._pool_timeout,
# max_retries=retry_strategy,
)
session = Session()
session.mount("http+unix://", unix_adapter)
return session
def send_request(
self,
session: Session,
method: str,
url: str,
headers: Optional[Headers] = None,
params: Optional[MutableMapping[str, str]] = None,
data: Union[str, MultipartEncoder, None] = None,
auth: Optional[Tuple[str, str]] = None,
) -> Response:
"""Send an HTTP request.
:param session: Requests session object.
:type session: requests.Session
:param method: HTTP method in lowercase (e.g. "post").
:type method: str
:param url: Request URL.
:type url: str
:param headers: Request headers.
:type headers: dict
:param params: URL (query) parameters.
:type params: dict
:param data: Request payload.
:type data: str | MultipartEncoder | None
:param auth: Username and password.
:type auth: tuple
:returns: HTTP response.
:rtype: arango.response.Response
"""
url_path = urlparse(url).path
unix_url = f"http+unix://{self.unix_socket_url}/{url_path.lstrip('/')}"
response = session.request(
method=method,
url=unix_url,
params=params,
data=data,
headers=headers,
auth=auth,
timeout=self.request_timeout,
)
return Response(
method=method,
url=response.url,
headers=response.headers,
status_code=response.status_code,
status_text=response.reason,
raw_body=response.text,
)
from arango import ArangoClient
# Instantiate the custom Unix socket HTTP client
unix_socket_http_client = UnixSocketsHTTPClient('path/to/unix/socket')
# Create an ArangoDB client instance with the custom HTTP client
db = ArangoClient(http_client=unix_socket_http_client).db(verify=True)
from python-arango.
@aMahanna Wow. Didn't expect such a positive and detailed answer. Thank you!
I found the http_client
immediately and I was already subclassing DefaultHTTPClient
. I also found requests-unixsocket
while digging in the issues of requests
.
But I had concerns about stability. What about timeouts? What about multithreading?
An official and tested solution would be nice. One that has been tested and used by many people.
Don't get me wrong. You code will probably work and I appreciate it. But it's new and requests-unixsocket
is not heavily used too.
I know it's complicated for this project by mostly being dependent of the requests
package.
That's basically ok because I trust requests
to be grown up and stable. python-arango
simply uses requests without any hacky ways so I trust it too. And python-arango
has probably a lot of use cases in the world already.
If just requests
would allow Unix sockets we had no problems here and other packages would benefit too.
Using Unix sockets is just not common or well known enough. What about making it official for python-arango
?
For testing and contributing to the community, I would be in!
from python-arango.
Hi @sausix,
Our primary aim with this driver is to maintain platform independence to the greatest extent possible. While Unix domain sockets offer advantages in terms of efficiency for inter-process communication, they are inherently platform-specific. Adding them would introduce a feature that is not natively supported on Windows environments.
The fact that Unix domain sockets are designed for inter-process communication on the same host makes it a relatively niche requirement for our driver. Not saying it's not useful (could be great for testing, for example), but it's way more common for the driver and the database server to be running on different hosts.
In our experience, very specific implementations that are not widely used by a significant portion of our user base can become dusty and challenging to maintain over time ("rotting code").
Don't get me wrong, I'm all for people using Unix domain sockets on top of our driver. We strive to make it as customizable and flexible as possible. I believe the approach suggested by @aMahanna , which involves extending the driver for specific needs, is the most practical solution. This way, we can keep the core driver platform-agnostic while still providing the means for users like yourself to implement more specific functionalities.
I would tend towards closing this as "Won't fix", but if you have any further thoughts or ideas, please feel free to share them with us.
from python-arango.
Hey @apetenchea
I fully agree with your arguments. Except running a database should imply it's primarily run on Linux hosts. Maybe there's a statistics? My personal use case will be Docker.
Don't want to argue anymore because I basically fully understand and agree to not break or fork for platform compatibility at too much cost.
I would totally be fine if we close this issue but at least keep @aMahanna 's wonderful snippet anywhere. Maybe here in the docs: HTTP Clients?
Could be marked as experimental and unmaintained. Using sockets would be a good real world use case for an HTTPClient
implementation.
I would definitely share my experience by pull requests if Unix sockets would be a seperate docs page for example.
Thank you.
from python-arango.
Related Issues (20)
- Documentation issue: ordered/geoJSON on collection.add_geo_index HOT 1
- Authenticate using User JWT-Token HOT 1
- Custom JSON serializer not being used for bulk operations HOT 3
- Edges not retrievable without committing transaction HOT 2
- python-arango is not working without pip-install installation HOT 3
- Content in API docs of branch `main` disappeared HOT 1
- merge functionality HOT 1
- Replacement for `pkg_resources` HOT 1
- Transactions Failing on Collection Deletion HOT 3
- Unable to disable request timeout HOT 2
- Requests with endpoint="/_api/simple/..." fail HOT 10
- API Specification Page is not rendering
- Collection.find does not work in 7.6.1 if filter attribute is set to none HOT 3
- Enum can't be serialized anymore after upgrade to 7.6.2 HOT 2
- bind_vars type hints are incomplete HOT 4
- broken build_filter_conditions for keys with spaces // backtick escaping missing? HOT 2
- Make ArangoClient generic to support typing when a custom serialiser is used HOT 9
- feature request: Continue an existing transaction HOT 3
- `arango.exceptions.GraphTraverseError: [HTTP 404][ERR 404] unknown path '/_api/traversal'` when using `traverse` (arangodb 3.12) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-arango.