Git Product home page Git Product logo

Comments (5)

apetenchea avatar apetenchea commented on June 20, 2024 1

Definitely, we should keep @aMahanna's example! It's already available on this issue, which is going to be indexed by search engines, since the project is public. People will be able to get here through a search.

The HTTP Clients page already contains a generic example. We should keep it as simple as possible, therefore I would be hesitant to add the UnixSocketsHTTPClient on top of that.

Nevertheless, you are more than welcome to share your expertise with our community. I'll be moving this thread into a "Show and tell" discussion here.
Please feel free share more info, gists or any relevant code snippets. Or, if you have a blog post where you wrote in more detail about Unix sockets in the context of ArangoDB, please do share it. All contributions are much appreciated!

from python-arango.

aMahanna avatar aMahanna commented on June 20, 2024

Hey @sausix!

This should be possible given that the ArangoClient constructor supports user-defined HTTP Clients, so we might be able to get away with creating a UnixSocketsHTTPClient that overrides the create_session() and send_request() methods to handle a Unix connection.

Although not natively supported in requests, you could consider the requests-unixsocket package as a workaround.

Here is what the DefaultHTTPClient in python-arango looks like, and here is what I can suggest as a draft for the UnixSocketsHTTPClient:

from typing import MutableMapping, Optional, Tuple, Union
from urllib.parse import urlparseimport requests_unixsocket
from requests import Session
from requests_toolbelt import MultipartEncoderfrom arango.http import DefaultHTTPClient
from arango.response import Response
from arango.typings import HeadersDEFAULT_REQUEST_TIMEOUT = 60
​
​
class UnixSocketsHTTPClient(DefaultHTTPClient):
    """Unix Socket HTTP client implementation.

    :param unix_socket_url: Unix socket URL.
    :type unix_socket_url: str
    :param request_timeout: Timeout in seconds for each individual connection.
    :type request_timeout: int | float
    :param retry_attempts: Number of retry attempts.
    :type retry_attempts: int
    :param backoff_factor: Backoff factor for retry attempts.
    :type backoff_factor: float
    :param pool_connections: The number of urllib3 connection pools to cache.
    :type pool_connections: int
    :param pool_maxsize: The maximum number of connections to save in the pool.
    :type pool_maxsize: int
    :param pool_timeout: If set, then the pool will be set to block=True,
        and requests will block for pool_timeout seconds and raise
        EmptyPoolError if no connection is available within the time period.
    :type pool_timeout: int | float | None
    """def __init__(
        self,
        unix_socket_url: str,
        request_timeout: Union[int, float, None] = DEFAULT_REQUEST_TIMEOUT,
        retry_attempts: int = 3,
        backoff_factor: float = 1.0,
        pool_connections: int = 10,
        pool_maxsize: int = 10,
        pool_timeout: Union[int, float, None] = None,
    ) -> None:
        self.unix_socket_url = unix_socket_url
        self.request_timeout = request_timeout
        self._retry_attempts = retry_attempts
        self._backoff_factor = backoff_factor
        self._pool_connections = pool_connections
        self._pool_maxsize = pool_maxsize
        self._pool_timeout = pool_timeoutdef create_session(self, host: str) -> Session:
        """Create and return a new session/connection.

        :param host: ArangoDB host URL.
        :type host: str
        :returns: requests session object
        :rtype: requests.Session
        """
        # retry_strategy = Retry(
        #     total=self._retry_attempts,
        #     backoff_factor=self._backoff_factor,
        #     status_forcelist=[429, 500, 502, 503, 504],
        #     allowed_methods=["HEAD", "GET", "OPTIONS"],
        # )unix_adapter = requests_unixsocket.UnixAdapter(
            self.request_timeout,
            self._pool_connections,
            # connection_timeout=self.request_timeout,
            # pool_maxsize=self._pool_maxsize,
            # pool_timeout=self._pool_timeout,
            # max_retries=retry_strategy,
        )
​
        session = Session()
        session.mount("http+unix://", unix_adapter)
​
        return sessiondef send_request(
        self,
        session: Session,
        method: str,
        url: str,
        headers: Optional[Headers] = None,
        params: Optional[MutableMapping[str, str]] = None,
        data: Union[str, MultipartEncoder, None] = None,
        auth: Optional[Tuple[str, str]] = None,
    ) -> Response:
        """Send an HTTP request.

        :param session: Requests session object.
        :type session: requests.Session
        :param method: HTTP method in lowercase (e.g. "post").
        :type method: str
        :param url: Request URL.
        :type url: str
        :param headers: Request headers.
        :type headers: dict
        :param params: URL (query) parameters.
        :type params: dict
        :param data: Request payload.
        :type data: str | MultipartEncoder | None
        :param auth: Username and password.
        :type auth: tuple
        :returns: HTTP response.
        :rtype: arango.response.Response
        """
        url_path = urlparse(url).path
        unix_url = f"http+unix://{self.unix_socket_url}/{url_path.lstrip('/')}"response = session.request(
            method=method,
            url=unix_url,
            params=params,
            data=data,
            headers=headers,
            auth=auth,
            timeout=self.request_timeout,
        )
        return Response(
            method=method,
            url=response.url,
            headers=response.headers,
            status_code=response.status_code,
            status_text=response.reason,
            raw_body=response.text,
        )
​
from arango import ArangoClient# Instantiate the custom Unix socket HTTP client
unix_socket_http_client = UnixSocketsHTTPClient('path/to/unix/socket')
​
# Create an ArangoDB client instance with the custom HTTP client
db = ArangoClient(http_client=unix_socket_http_client).db(verify=True)

from python-arango.

sausix avatar sausix commented on June 20, 2024

@aMahanna Wow. Didn't expect such a positive and detailed answer. Thank you!

I found the http_client immediately and I was already subclassing DefaultHTTPClient. I also found requests-unixsocket while digging in the issues of requests.
But I had concerns about stability. What about timeouts? What about multithreading?
An official and tested solution would be nice. One that has been tested and used by many people.
Don't get me wrong. You code will probably work and I appreciate it. But it's new and requests-unixsocket is not heavily used too.

I know it's complicated for this project by mostly being dependent of the requests package.
That's basically ok because I trust requests to be grown up and stable. python-arango simply uses requests without any hacky ways so I trust it too. And python-arango has probably a lot of use cases in the world already.

If just requests would allow Unix sockets we had no problems here and other packages would benefit too.

Using Unix sockets is just not common or well known enough. What about making it official for python-arango?
For testing and contributing to the community, I would be in!

from python-arango.

apetenchea avatar apetenchea commented on June 20, 2024

Hi @sausix,

Our primary aim with this driver is to maintain platform independence to the greatest extent possible. While Unix domain sockets offer advantages in terms of efficiency for inter-process communication, they are inherently platform-specific. Adding them would introduce a feature that is not natively supported on Windows environments.
The fact that Unix domain sockets are designed for inter-process communication on the same host makes it a relatively niche requirement for our driver. Not saying it's not useful (could be great for testing, for example), but it's way more common for the driver and the database server to be running on different hosts.
In our experience, very specific implementations that are not widely used by a significant portion of our user base can become dusty and challenging to maintain over time ("rotting code").

Don't get me wrong, I'm all for people using Unix domain sockets on top of our driver. We strive to make it as customizable and flexible as possible. I believe the approach suggested by @aMahanna , which involves extending the driver for specific needs, is the most practical solution. This way, we can keep the core driver platform-agnostic while still providing the means for users like yourself to implement more specific functionalities.

I would tend towards closing this as "Won't fix", but if you have any further thoughts or ideas, please feel free to share them with us.

from python-arango.

sausix avatar sausix commented on June 20, 2024

Hey @apetenchea

I fully agree with your arguments. Except running a database should imply it's primarily run on Linux hosts. Maybe there's a statistics? My personal use case will be Docker.
Don't want to argue anymore because I basically fully understand and agree to not break or fork for platform compatibility at too much cost.

I would totally be fine if we close this issue but at least keep @aMahanna 's wonderful snippet anywhere. Maybe here in the docs: HTTP Clients?
Could be marked as experimental and unmaintained. Using sockets would be a good real world use case for an HTTPClient implementation.
I would definitely share my experience by pull requests if Unix sockets would be a seperate docs page for example.

Thank you.

from python-arango.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.