Is it possible to use `argparse` together with absl? about abseil-py HOT 11 CLOSED

This is now available

from abseil-py.

Update:

We are now considering to provide a absl.flags.argparse_flags module to let users use argparse together with absl.

The module provides a argparse_flags.ArgumentParser class. It has the same API as argparse.ArgumentParser, but will take into account all flags defined with absl.flags.

And to use it together with absl.app.run, you will need to define your argparse flags and call parse_args inside a function with type Callable[[List[Text]], Any]:

def parse_flags(argv):
    parser = argparse_flags.ArgumentParser(description='An argparse + app.run example')
    # Add your argparse flags.
    return parser.parse_args(argv[1:])  # The return value is passed to `main` function by `app.run`.

Then pass the function to app.run:

app.run(main, flags_parser=parse_flags)

Here is a full example:

from absl import app
from absl import flags
from absl.flags import argparse_flags

FLAGS = flags.FLAGS
flags.DEFINE_string('echo', 'Hello', 'Message to echo.')


def parse_flags(argv):
    parser = argparse_flags.ArgumentParser(description='An argparse + app.run example')
    parser.add_argument('--header', help='Header message to print.')
    return parser.parse_args(argv[1:])


def main(args):
    if args.header:
        print('----', args.header, '----')
    print('Echo:', FLAGS.echo)

if __name__ == '__main__':
    app.run(main, flags_parser=parse_flags)

You can use the program such as:

./program --header example --echo 'Hello argparse'

Thoughts? Feedbacks?

from abseil-py.

Since absl.flags can be defined in any module, there isn't an ordering of when they are defined. So we can't change them to be positional. Those must be flags, i.e. specified with --echo=value or --echo value.

However, you will be able to define positional arguments using argparse:

parser = argparse_flags.ArgumentParser(description='An argparse + app.run example')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
                    help='an integer for the accumulator')

You will also be able to use the argparse sub-parsers. Here is an example:

flags.DEFINE_string('echo', 'Hello', 'Message to echo.')

parser = argparse_flags.ArgumentParser(description='Sub parser example.',)
subparsers = parser.add_subparsers(help='The command to execute.')

roll_dice_parser = subparsers.add_parser(
    'roll_dice', help='Roll a dice.',
    # Do not make absl FLAGS available after the subcommand `roll_dice`.
    inherited_absl_flags=None)
roll_dice_parser.add_argument('--num_faces', type=int, default=6)

shuffle_parser = subparsers.add_parser('shuffle', help='Shuffle inputs.')
shuffle_parser.add_argument(
    'inputs', metavar='I', nargs='+', help='Inputs to shuffle.')

By default, absl.flags can be specified before or after the positional sub-command. Use inherited_absl_flags=None to only allow before (the roll_dice sub-command in the example).

Usage:

./program --echo=value roll_dice --num_faces 20
./program shuffle 1 2 3 4 --echo=value

Does this meet your lab's demand?

from abseil-py.

Thanks @raymondchua! I updated my previous comments.

from abseil-py.

If you don't need to change these flags' values in absl.logging in the command line, you should be able to call absl.logging just fine, with one tweak: call absl.flags.FLAGS.mark_as_parsed() first, otherwise the first log will emit a warning to stderr here.

You could also programmatically set these flag values like this:

FLAGS.stderrthreshold = 'warning'

If you do want command line flags to affect absl.logging, there isn't a way to magically blend argparse flags and absl.flags flags. There are a couple of solutions.

1. Let one parser parse "known flags", and the other parse the rest.

Example 1:

import argparse
import sys
from absl import flags

def main():
  parser = argparse.ArgumentParser()
  parser.add_argument('--foo', help='foo help')
  args, unknown_flags = parser.parse_known_args()  # Let argparse parse known flags from sys.argv.
  flags.FLAGS(sys.argv[:1] + unknown_flags)  # Let absl.flags parse the rest.

  # Main program starts here.

if __name__ == '__main__':
  main()

Example 2:

import argparse
import sys
from absl import flags

def main():
  unparsed = flags.FLAGS(sys.argv, known_only=True)  # Let absl.flags parse known flags from sys.argv.
  parser = argparse.ArgumentParser()
  parser.add_argument('--foo', help='foo help')
  args = parser.parse_args(unparsed[1:])  # Let argparse parse the rest of flags.

  # Main program starts here.

if __name__ == '__main__':
  main()

The caveat is that the (future) code might define same flags in both argparse and absl.flags, and such bugs can be silently introduced. So I don't recommend this approach.

2. Explicitly separate flags in the command line.

Example 1:

import argparse
import sys
from absl import app
from absl import flags

def main(argv):
  parser = argparse.ArgumentParser()
  parser.add_argument('--foo', help='foo help')
  args = parser.parse_args(argv[1:])  # Let argparse parse the rest of flags.

  # Main program starts here.

if __name__ == '__main__':
  app.run(main)

This code requires an extra -- in the command line, e.g. ./program --alsologtostderr -- --foo bar. (absl.flags treats everything after the first -- as positional arguments, and app.run passes them to the main function.)

Example 2:

import argparse
import sys
from absl import flags

def main():
  parser = argparse.ArgumentParser()
  parser.add_argument('--absl_flags', action='append', default=[], help='absl flags')  # Use --absl_flags to specify absl.flags.
  parser.add_argument('--foo', help='foo help')
  args = parser.parse_args()
  flags.FLAGS(sys.argv[:1] + args.absl_flags)

  # Main program starts here.

if __name__ == '__main__':
  main()

This code requires the absl.flags flags to be specified w/ --absl_flags, e.g. ./program --absl_flags=--alsologtostderr --foo bar --absl_flags=--stderrthreshold=warning.

Let me know if this helps.

from abseil-py.

Thanks a lot! I will try it sometime and update you soon.

from abseil-py.

Hi, @yilei , thanks for the comment. It works for me.

But one caveat to point out is that in this way we cannot use absl.app to run the program unless we explicitly separate the flags. For example, the logging verbosity will not be bumped to INFO if we do not use absl.app.run to start the program.

from abseil-py.

Right, I forgot that. You can also explicitly set the verbosity with logging.set_verbosity.

from abseil-py.

Hi, @yilei,

Thanks for the update. It is definitely a great feature for outside users!

A bit more comments, I think, for our lab, the most demanded features are the positional arguments and the sub-parsers in argparse packages. It might be even better to integrate those features into absl.flags. To be a bit more specific, I am expecting something like

flags.DEFINE_string("echo", "hello", "Message to echo.", positional=True)

from abseil-py.

Hi, @yilei,
Thanks. It is absolutely much closer to what we need.

from abseil-py.

I am on absl-py==1.0.0 and I have to make the following change.

def parse_flags(argv):
    parser = argparse_flags.ArgumentParser('An argparse + app.run example')
    parser.add_argument('--header', help='Header message to print.')
    return parser.parse_args(argv[1:])

The code above doesn't work for me but this does:

def parse_flags(argv):
    parser = argparse_flags.ArgumentParser( description='An argparse + app.run example')
    parser.add_argument('--header', help='Header message to print.')
    return parser.parse_args(argv[1:])

Looks like the keyword description is needed.

from abseil-py.