mysteryblokhed / databind Goto Github PK
View Code? Open in Web Editor NEWExpand the functionality of Minecraft Datapacks.
Home Page: https://databind.rtfd.io/en/stable
License: GNU General Public License v3.0
Expand the functionality of Minecraft Datapacks.
Home Page: https://databind.rtfd.io/en/stable
License: GNU General Public License v3.0
Right now, a function might look like this:
func example
say Hello, World!
end
It would be easier to tell what's where if tabbing in lines inside functions was recommended, like this:
func example
say Hello, World!
end
Or this:
func example
var i := 10
while tvar i matches 1..
var i -= 1
end
end
Documentation examples and test .databind
files should also be changed.
Function calls are outputted with forward slashes prepended to them (eg. :call function
becomes function namespace:/function
). This is most likely a problem with get_subfolder_prefix
in main.rs
. The line :call function
should become function namespace:function
, without the prepended slash.
Databind should be able to run properly on Windows, macOS, and Linux distributions without problems. Adding tests that properly check the output of files instead of just that they're there would also help ensure that output doesn't vary on different operating systems.
The vars.ini
file added in #92 lets developers define variables that can be used across the program in an easily configurable way. However, the .ini
filetype isn't meant for this and doesn't support some features that might be useful, like multiline strings or other types. Using an .ini
file also means extra dependencies.
Switching over to a .toml
file would mean using existing dependencies and would also add support for more types. These types would be converted to strings, but it would mean a better-looking file better suited for something like this.
Right now, the integration tests are set up via a bunch of files that each contain only a couple of functions. Moving these test functions into files with more general purposes (eg. file_structure_tests.rs
and file_contents_tests.rs
) would mean less files in the tests folder and a more clear idea of where to place new tests.
Minecraft's scoreboard players operation
command has more functionality than the current assignment operators that Databind has. Changing these assignment operators to use the scoreboard operations would make Databind variables more useful and improve the objective shorthand.
This settings hasn't been updated for almost every required feature of Databind. It's likely that someone might try to use it, see that it doesn't work, and thing that either the entire project doesn't work or that the feature is broken (and they'd be right). Removing it altogether instead of trying to update it would probably be easiest at this point.
The word "transpile" is used both in the source code for function/file names and in the documentation. More people are probably familiar with the word "compile" and would recognize it easier than "transpile." Renaming files, structs, functions, etc. and updating the docs to use "compile" might be a good idea.
When running databind
with no args in a project, the output folder is generated relative to the directory the command was run in, as opposed to the project root.
I'll add more information if I find anything that might help.
The ability to define multiline replacements with :def
would make it a lot more useful. For example, if you want to make a file that lets people configure your pack by defining functions, having a definition that inserts a few lines to set up the function. Something like this:
:mdef MAKE_VARS
:var var1 .= 0
:var var2 .= 1
:enddef
Replacement should still work as single-line replacements do, replacing any occurrence of the replacement name (in this case MAKE_VARS
).
Functions tagged in Databind using the tag
keyword are added to a HashMap that's turned into tag files at the end of compilation. The problem with this is that any function tags that already contained things will be overwritten. Since the project already uses serde
and serde_json
, it wouldn't be hard to check if each tag .json
file already exists and, if it does, add it to the TagFile
struct created at the end.
I've marked this issue as a bug because Databind is meant to be able to be implemented alongside normal mcfunctions, or to be gradually integrated into a project.
Support for text replacement in a similar manner to C language #define
's would be nice to have for repetitive commands or portions of commands.
For example, if you frequently use a command like execute as @a[scores={example_obj=1..},nbt=
etc., then you could define a replacement and reuse that instead. In addition to making long lines shorter, it would also make it easier to change one part of the command. Instead of trying to change it everywhere, making it possible to accidentally miss a line and create a bug, you only have to change one line at the top of your file.
The syntax could look something like this:
:def LONG_EXECUTE execute as @a[scores={custom_item_obj=1..},nbt={SelectedItem:{id:"minecraft:carrot_on_a_stick",tag:{custom_item:1b}}}] at @s
:func custom_item_stuff
LONG_EXECUTE run summon minecraft:lightning_bolt
:endfunc
The code for while loops is already effectively a preprocessor, so that code could be reused or reformatted to include finding text replacement definitions.
Shorthand to delete objectives could be useful in functions that "uninstall" a datapack and could benifit from a shorter way to remove objectives they've created. Adding one or both of the keywords listed in the issue title (or something similarly named) could help to solve this.
These keywords would both do the same thing, so it would be up to preference which one to use. In practice, it would look something like this:
:delvar var_or_objective1
:delobj var_or_objective2
compiles to:
scoreboard objectives remove var_or_objective1
scoreboard objectives remove var_or_objective2
The text replacement definitions can be useful, but adding Rust-like macros could also be useful. If you want to make a pack that lets users define their own functions to add functionality but you don't want to make them copy+paste a bunch of boilerplate, then you could use a macro to do it for them. Something like this could work:
!macro on_event
!takes $event
!takes $name
!takes $run
func $name
tag tick
obj $name $event
execute as @e[scores={$name=1..}] run $run
endfunc
!end
The above could be used as something like:
?on_event minecraft.used:minecraft.carrot_on_a_stick item_used say Hello!
The macro call would be converted to the following Databind code:
func item_used
tag tick
obj item_used minecraft.used:minecraft.carrot_on_a_stick
execute as @e[scores={item_used=1..}] run say Hello!
endfunc
The syntax is just an idea, so there might be better ways to implement it.
If this issue is closed, !def
definitions should be removed.
The transpile function returns the transpiled contents of the .databind
file, even though Databind files are now only meant to contain function definitions. The main function throws away the contents of the default file, so it doesn't need to be returned.
Currently, you are unable to nest macro definitions like you can with if statements or while loops. The following definition...
!def macro_1($name)
!def macro_2_$name()
!end # End 1
!end # End 2
...would result in macro_1
being ended prematurely due to the line marked End 1
. Since this functionality is undocumented, it should be considered a bug.
If support for nested definitions like this is added, then it the compiler would need to be updated to prioritize parsing of macro definitions before anything else. In the following situation:
!def macro_1($name)
!def macro_2_$name()
say Example
!end
!end
func main
?macro_1("test")
?macro_2_test()
end
The macro macro_2_test
will not have been defined yet.
I'm not sure of the best way to fix this right now.
Something similar to a while loop is already possible using functions (can be seen in the loop function example), but it can get a bit messy with a lot of similarly-named functions and variables. Creating syntax for a while loop could make things easier. Something like this could work:
:var i .= 5
:while :tvar i matches 1..
say Still in loop!
:var i -= 1
:endwhile
say i is 0
With currently implemented databind functions, this would be roughly equivalent to this:
:var i .= 5
:func main_loop
execute if :tvar i matches 1.. :call loop_condition
:endfunc
:func loop_condition
say Still in loop!
:var i -= 1
:endfunc
:call main_loop
Datapacks support tags by creating a /tags folder under a namespace folder. These tags are referenced using a #
. For example:
kill @e[type=#namespace:my_tag]
The support for comments means that this like in a .databind
file would be compiled to:
kill @e[type=
The tag and anything following it is removed since the compiler thinks it's a comment. Comments should only be possible if the #
is at the beginning of a line, not partway through.
Databind is currently missing integration tests for while loops and replacement definitions. Adding integration tests for these will make sure that no code accidentally breaks a feature and goes unnoticed for some time.
WIth the current --generate-func-tags
argument, function tags in minecraft/tags/functions
are generated for functions specified in the config, or the default functions (load
and tick
). However, it's not currently possible to have multiple functions in one tag with this setup. It should be possible to tag functions in-code instead of having to manually create the tag files when using multiple namespaces or multiple functions that should be tagged.
The syntax could look something like this:
:func example :tag load :tag secondTag
# etc
:endfunc
or:
:func example
:tag load
:tag secondTag
# etc
:endfunc
The second would probably be better since it would allow multiple tags on one function without creating long lines.
Due to the implementation of functions, there cannot be nested functions even if they would not be nested in the output. This also means that while loops cannot be used in functions defined with :func
, since a while loop becomes two functions.
Replacing the in_function
boolean in transpile.rs
with an integer that counts the current depth could help fix this problem. The current_function
variable would also need to be changed to a Vec<String>
of function names instead of just the name of the current function.
Add support for a TOML configuration file that lets you configure what/how Databind transpiles.
It should look something like this:
databind.toml
random_var_names = false
var_display_names = false
# Whether to auto-generate function JSON in minecraft/tags/functions
generate_func_json = true
# Functions not to generate JSON for
func_json_exclusions = [
"main",
"loop1"
]
Any values that aren't lists should also be passable as CLI arguments. An optional --config
/-c
argument should be added to specify the location of a config file. If the argument is not passed, Databind should look for a config file in the directory it's being run in. Otherwise, use default settings.
I accidentally merged #84 without properly testing the code in-game, and just found out that they don't actually work for a couple of reasons.
I'll try to fix this as soon as I get the chance.
The config file's output is relative to where databind
was run as opposed to where the config file is. This can cause problems when trying to test the config file's output. Output should be relative to the config file regardless of whether the config file has an output folder specified.
It's a bit weird to have some .databind
files that generate their own functions and some that don't. It would probably be best to remove the function_out_exclusions
config option and make what it does default for all files. This would mean that every .databind
file should only contain function definitions.
This behavior makes the most sense since tagging can only be done in function definitions. Another option would be to keep the current system but to allow :tag
s at the top of files, but the function_out_exclusions
config option can be confusing and is unlike any other language.
Documentation mentioning function_out_exclusions
should also be updated.
The transpile function currently returns an enum called TranspileReturn
. With single-file compilation being removed, this is no longer necessary. It should only return the equivalent of what is currently called MultiFileAndMap
.
Due to the documentation's use of sphinx.ext.autosectionlabel
, some labels like example
or compiled
are reused many times, creating warnings. While these warnings won't actually affect anything, they could detract from actual warnings or errors if they fill the output with warnings.
Anywhere with duplicate labels should be updated to have unique ones related to the docs page (eg. example
to simple-function-example
in simple_function.rst
).
The compile()
function has to do some extra work when it comes to tokens such as Token::VarName
. It could mean that a variable is being modified, returning a test for an execute if
, being deleted, or, when #45 is closed, returning something for a scoreboard player operation. Having unique tokens for each of these cases (eg. ModVarName
, TestVarName
, DelVarName
, and OpVarName
) would meant that the compiler doesn't have to look at the last token to find out what it should do with the variable name it's given.
The ability to compile single files is likely not going to be used much as single .mcfunction
files are not very useful on their own. It also means having to document additional features and making sure that new features are supported for single-file compilation.
Documentation mentioning single-file compilation should also be updated.
A keyword similar to :call
for predicates and tags would be a good addition since they use a similar syntax of namespace:thing
, such as:
execute if predicate namespace:predicate
execute as @a[predicate=namespace:prediate]
or:
kill @e[type=#namespace:tag]
Support for tags or the second form of predicate might be harder due to the way that Databind tokenizes by spaces, but predicates should be possible just by copy+pasting and tweaking the code for :call
.
Pull request #7 added a :tag
keyword and removed the generate_func_tags
config option. Most documentation examples still use files named load.databind
and tick.databind
, but these files are no longer automatically tagged. Examples should be replaced with single main.databind
files using functions tagged with :tag
, like this:
example/data/example/functions/main.databind
:func load :tag load
tellraw @a "Loaded"
:endfunc
:func tick :tag tick
tellraw @a "Ticking"
:endfunc
Transpiled output should be able to stay the same, assuming that the function names stay the same.
The tokenize()
function contains many copies of this code:
if self.current_char.is_whitespace() {
tokens.push(Token::VarName(current_keyword));
building_keyword = false;
building_token = Token::None;
current_keyword = String::new();
first_whitespace = false;
} else {
current_keyword.push(self.current_char);
}
Moving this to another function that takes a token as an argument would make the code cleaner. It would also make it much easier to change how tokens are added, since it'd be in one place instead of as many places as tokens.
Any other common copy+pasted code throughout the project could also be moved to a new function in the same file.
An if/else statement:
func condition
tag load
var i := 10
if tvar i matches ..9
say var i is less than 9
var i = 10
say switched var i to 10!
else
say var i is 10 or above
var i = 10
say switched var i to 10!
end
end
This would be compiled similar to a while loop, creating functions to run the if, or else.
function example:condition
execute if score example matches ..9 run function example:if_abcd
execute unless score exaple matches ..9 run function example:else_abcd
function example:if_abcd:
say var i is less than 9
scoreboard players set example i 10
say switched var i to 10!
function example:unless_abcd:
say var i is 10 or above
scoreboard players set example i 10
say switched var i to 10!
When running with either the --generate-func-tags
flag or generate_func_tags = true
in config, even with an empty inclusions list, everything is transpiled properly. However, if either the flag is missing or the config setting is false, all of the files will be put directly in the <project>.databind
folder instead of a data subfolder.
I am unsure why this is happening right now.
After #54 is closed, it would be worth considering removing the :
prefix from all keywords. It was initially added to make tokenization easier, but the rewritten tokenizer won't need the prefix to identify keywords. Removing the colon would also make Databind look more like a regular language instead of something foreign. Documentation would also need to be updated.
If you want to perform a scoreboard operation on two databind variables, you'd currently have to do it like this:
scoreboard players operation --databind var1 = --databind var2
It would be easier to have a keyword that becomes --databind var1
or --databind var2
. The :tvar
keyword almost does this, but it also has score
at the beginning (eg. :tvar var1
-> score --databind var1
). Adding a new keyword like :gvar
to get just what you'd need to perform an operation would be a good idea. That way, the above could be written as such:
scoreboard players operation :gvar var1 = :gvar var2
Adding another keyword like :varop
or something that becomes scoreboard players operation
would shorten the above even further to the following:
:varop :gvar var1 = :gvar var2
One main issue to track things that are very broken.
databind
with no arguments, the output folder isnamespace:/funcname
instead of namespace:funcname
).databind
files seem to stop being transpiled after a while loop, and the while loopFor some reason, having two or more !def
lines causes all but the first not to work. The !def
lines are excluded from the output, but the text is not replaced. For example, this code:
!def t1 test1
!def t2 test2
func main
say t1
say t2
end
Compiles to:
say test1
say t2
Since the tokenizer will find any keyword no matter the context and try to tokenize it, it's impossible to use the keywords anywhere else, even in tellraw
or say
commands. Adding an escape character that escapes an entire keyword would help solve this problem, like this:
call func1
say ^call func1
which would become:
function namespace:func1
say call func1
If a developer wanted to use the escape character at the beginning of a word, they could just put two of them:
say ^^example
which would become:
say ^example
When trying to use global macros with the files !macros.databind
and main.databind
on Linux, the error A non-existant macro test was called
was printed. It seems like files with !
at the beginning either aren't being tokenized first, or aren't being recognized as global macros. The files in question are from PR #95 and can be seen here.
PR's to fix this should probably branch off of #95 to make sure that the new test is there.
Right now, the documentation has a few examples for a few features and a table with language syntax, but there's very little information on how to get started with a new project. There should be a "Getting Started" page that helps people set up the structure for a project and how to use databind with it. The file structure follows normal datapacks, but not everyone will know how to set one up.
Some things to do:
databind create
main.databind
versus other filesHaving a variables.toml
file or something similar to define global variables could make it easier to let users customize settings. An example of a variables.toml
file could look like this:
delay_in_ticks = 500
do_something = true
do_something_else = false
True or false values would be converted to 1 or 0.
The globals could then be referenced in a .databind
file by using some preceding character, like a %
:
func load
tag load
var do_something := %do_something
var do_something_else := %do_something_else
var delay := %delay_in_ticks
end
func tick
tag tick
execute if tvar do_something matches 1 run tellraw @a "do_something is true"
execute if tvar do_something_else matches 1 run tellraw @a "do_something_else is true"
end
Maybe a bit ambitious, but the ability to add functionality to Databind with extensions could make it applicable to more projects. If this is going to be done, it should probably be done when the language's syntax has become more stable.
I was thinking that using PyO3 to let people add functionality with Python would be easiest for most developers. This would also mean creating a new Rust library, adding a new codebase and more to maintain. This is likely not a good idea to do right now, since bugs are still being ironed out in the main project. But once most bugs are fixed, this could be a nice addition.
Every test starts with very similar code that looks something like this:
let mut path = tests::resources();
path.push("some_test_directory");
let out = TempDir::new("some_test_directory").expect("Could not create tempdir for test");
tests::run_with_args(
"cargo",
&[
"run",
"--",
path.to_str().unwrap(),
"--ignore-config",
"--out",
out.path().to_str().unwrap(),
],
None,
);
Since this code is used in basically every test, it would be a good idea to move the code to a function that returns out
and, if it's used (I can't remember if any tests use it or not), path
.
Currently there is a databind create
command that can be used to make a project, but it could be easier to build. The current system that the CLI uses to build is just to pass a path, for example databind ./my_project
. There are options for output, configuration, etc., but it would be nice to be able to just run databind
and have it build based on the configuration file.
The generation of the configuration file should be able to use the default values for the Settings struct, but the output directory should be specified instead of left blank. Building by passing a path to the CLI should still be allowed, but it should also look for a databind.toml
file in the target directory. Running databind
should work no matter the current directory of the command line. For example, it should work whether you're in the project's root, in the data folder, or in a functions folder.
The folder structure from databind create
could also use some improvement. It should look more like this:
project_name
│ databind.toml
└───src
│ pack.mcmeta
└───data
└───project_name
└───functions
main.databind
This way, the generated folder will also contain the output folder instead of generating a new one.
databind create
generate a databind.toml
with default settingsdatabind create
generate the aforementioned file structure<project_name>.databind
as the default output folder with out
or something similardatabind
alone to be run in a project and detect the config file
databind.toml
file no matter the current directory in the projectSome datpacks have subfolders under their namespaces to keep code more organized, like this:
mypack
│ pack.mcmeta
└───data
└───mypack
└───functions
│ main.mcfunction
└───cmds
a_command.mcfunction
This could look like this as a Databind project:
mypack
│ databind.toml
└───src
│ pack.mcmeta
└───data
└───mypack
└───functions
│ main.databind
└───cmds
commands.databind
This will successfully compile if the subfolders only contain untagged functions, do not contain calls to any other functions, and do not use while loops. However, this is unlikely.
You can already call a function from commands.databind
from main.databind
using :call cmds/func_name
, but you can't call main from commands.databind
.
Since this is already a feature of datapacks and Databind is mostly meant to be a superset of datapacks, this will be considered a bug.
Fixing function calls would also fix while loops due to the way they were implemented.
Right now, Databind will recompile every file in a project when built, even though it's unlikely that every file in a project has changed. In large projects, this can be a significant problem. I'm not certain how other incremental compilers make sure to only recompile changed files, but doing something like creating a .databind
folder in a project's root with the SHA256 hashes of files could work.
In an example implementation, the following tree:
src
└──data
└──namespace
└──functions
main.databind
would result in the following .databind
folder:
.databind
└──src
└──data
└──namespace
└──functions
main.databind.info
Before file's signatures are checked, the file modification date should be checked. There's no reason to check the hash of a file if it hasn't been modified. This could be stored in the same file as the hash, resulting in the following contents for main.databind.info
:
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 1626831659
Where e3b0c442...
is the file's hash, and 1626831659
is the file's modification time in seconds.
It would be nice to be able to start a new Databind project just from the CLI instead of having to set up the files and folders yourself. The generated folders/file should look like this:
project_name
├── data
│ └── project_name
│ └── functions
│ └── main.databind
└── pack.mcmeta
The command should be something like databind create --name <project_name> --description <description> --path <path>
. The only required arguments should be name. Description can default to nothing and the path should default to .
.
Right now, the tokenize()
function will look for a colon preceded by a space before attemting to turn something into a keyword, and anything else is put into a token called NonDatabind
. There are some problems with this system. In the way it's implemented, it's impossible for a token to start with another (eg. you couldn't have :def
and :deffunc
, since it would find def
and stop looking for new keywords). It's also impossible to make a token that doesn't begin with :
, which is pretty unusual for any language.
If the tokenizer was rewritten to just treat anything separated by whitespace as a separate token, then there would be a lot more freedom as to what tokens could be named, etc. Any space-separated tokens that weren't recognized by Databind would just be assumed to be normal Minecraft commands and kept as-is. However, this is not an issue for this PR. The following:
func test
say hi
endfunc
Could be tokenized as something like: [Token::DefineFunc, Token::FuncName("test"), Token::NewLine, Token::NonDatabind("say"), Token::NonDatabind("hi"), Token::NewLine, Token::EndFunc]
.
Right now, the tests for the contents of while loops have to use globs to find the correct files. If more than one while loop was to be used in a test, then it would take some extra code to find out which output file was generated by which loop. This problem gets more complicated with if/else statements (#84), since they generate more files and some files won't always exist.
Adding a CLI option to pass a list of chars to use instead (eg. --chars aaaa,bbbb
) could solve this problem, since the output files will always have the same characters at the end.
I think that Clap can deal with multiple values passed for the same argument, but I'm not sure if it's comma-separated, or if you pass the argument more than once (eg. --chars aaaa --chars bbbb
), or something else.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.