Comments (2)
I would like to propose major simplification of the current magic framework. Please find some details below.
Definition of Magic
class MagicProtocol(typing.Protocol):
def __call__(
data: Text,
*args:Union[Text, int, bool],
**kwargs:Union[Text, int, bool]
) -> Any:
pass
Definition of Argument parsers arguments
Argument types would be defined separately for each type. For example argument of type int would be defined as:
class MagicIntArgument:
flags: List[Text]
default: Optional[Int]
help: Text
def add_to_argparse(self, parser argparse.ArgumentParser):
# code to transform MagicIntArgument to argparse.ArgumentParser.add_argument invocation
...
Similar classes would be defined for other argument types (Text, bool and others as needed).
picatrix_magic
decorator signature would stays the same:
def picatrix_magic(
function: Optional[MagicProtocol] = None,
arguments: Optional[List[MagicArgument]] = None,
name_func: Optional[Callable[[Text], Text]] = None,
conditional: Optional[Callable[[None], bool]] = None
) -> MagicProtocol:
...
If arguments
is provided, it will be used to parse magic arguments and provide them to decorated function.
If it's not provided, arguments will get parsed out of decorated function definition
def _magic_to_arguments(magic: MagicProcotol) -> List[MagicArgument]:
hints = typing.get_typing_hints(magic)
...
Rest of the code will stay more or less the same, but will follow the path in which arguments
is provided - either explicitly by caller or parsed out of magic definition.
Concerns:
@bladyjoker, @kiddinn - I would like to ask your opinion on following topics:
- Magic return type
Any
type is a broad one and kind of defeats a purpose of having a type at all. Initially I was thinking about making itpd.DataFrame
, but then I saw that Timesketch magics are using Dict and other types as output. Do you see any way to define all possible valid outputs of the Magic? - Generic MagicIntArgument
Implementation that I proposed above is serving the purpose, but is very repetitive and inflexible. I think it can be improved through use oftyping.Generic
. Do you happen to have an experience with Python generics and can share your opinion here? Alternatively - can you suggest any environment where I can quickly check my assumptions abouttyping
? My usual development cycle is to validate assumptions in REPL and then code the most succesful solution. Unfortunately type verification is disabled in runtime. - I would like to use https://pypi.org/project/docstring-parser/ for parsing docstrings and extracting descriptions of arguments. In the future I would like to also use it to enable more end-user-friendly
functools.partial
s (reformatting docstrings to take into the account that some attributes got already applied by partial). Do you have any objections against this dependency?
from picatrix.
ok, to answer your questions here
-
Return type. It is hard to predict the future need of a return type, often times it is a DataFrame, but it can be a dict, it can be a single value, int, string, etc... it can also be none... as in
timesketch_set_active_sketch
which doesn't produce an output. You may want to write a magic that returns a bool to indicate whether it was successful or not, that's the reason for tehAny
, I agree it is not ideal, and it would be good to have something else, but I'm not sure what we can do here without restricting the magics to a limited set/usability. -
We need to start adding more typing verification... we can use tools like
mypy
to automatically run against the codebase (I've been experimenting with it, but I need to make some changes to have it work in the first place). This could run on every PR, just like pylint and unit tests, to make sure the codebase is enforcing typing. My idea is also to get rid of the docstring parsing toenforce
the typing and use the build-int typing to do the casting and enforcing of arguments. -
I don't have any objections about that.
I was also thinking, do we need to have the magic arguments at all, or just have the typing in the functions and parsing of docstrings? Do we need the added flexibility but still a lot more complexity by also supporting defining these MagicArguments?
from picatrix.
Related Issues (10)
- Add e2e tests for picatrix
- Contributing.md missing in project HOT 2
- Add a helper function registration
- Use typing to construct argument list
- Add magics for osquery
- Add magics to work with Timesketch
- Add magics that work with GRR HOT 1
- Add an initialization file
- Add unit tests for all files that are in
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from picatrix.