Scripting Swift

March 17, 2020

Federico Zanetello @zntfdr

The Swift team has recently announced ArgumentParser, a new parse command-line argument library.

In previous entries we've covered how we can parse command-line arguments manually, and with TSCUtility:

ArgumentParser comes with an extensive and very well written documentation, please give it a read for a complete overview of the library API.

With the basics covered, in this article we're going to dive into how things are implemented under the hood, like we did for the Swift Preview Package.

ParsableCommand Protocol

When following the instructions, the first task is create a type conforming to the ParsableCommand protocol:

public protocol ParsableCommand: ParsableArguments {
  static var configuration: CommandConfiguration { get }
  static var _commandName: String { get }
  func run() throws
}

Snippet from ArgumentParser's ParsableCommand.swift.

_commandName, which represents our tool name, is marked as internal: it's not for us to implement.

All these methods are implemented for us already, in our type we only need to take care of any custom logic: mainly the run() behavior and, if necessary, extra configurations such as abstract and subcommand declarations. Here is the default implementation:

extension ParsableCommand {
  public static var _commandName: String {
    configuration.commandName ??
      String(describing: Self.self).convertedToSnakeCase(separator: "-")
  }
  
  public static var configuration: CommandConfiguration {
    CommandConfiguration()
  }
  
  public func run() throws {
    throw CleanExit.helpRequest(self)
  }
}

Snippet from ArgumentParser's ParsableCommand.swift

In this default implementation we discover the first magic trick used in ArgumentParser:
how our type name transforms from camelCase into a kebab-case command line tool name:

String(describing: Self.self).convertedToSnakeCase(separator: "-")

Self refers to the type conforming to the protocol, and using String(describing: Self.self) (or String(describing: self), since _commandName is a static property) will return us our type name as a String.

Once we have the name, ArgumentParser has a convertedToSnakeCase(separator:) function that takes care of the rest.

ParsableArguments Protocol

The ParsableArguments protocol requires our types to conform to the ParsableArguments protocol as well, let's take a look at that next:

public protocol ParsableArguments: Decodable {
  init()
  mutating func validate() throws
}

Snippet from ArgumentParser's ParsableArguments.swift

Its definition is A type that can be parsed from a program's command-line arguments, which also explains the required Decodable conformation:
the library has its own decoders, ArgumentDecoder and SingleValueDecoder, which are used later on, to decode the command-line arguments into something meaningful.

Both the initializer and the Decodable conformation are synthesized, the only requirement left is the validate() implementation, however the library offers a default implementation for us already:

extension ParsableArguments {
  public mutating func validate() throws {}
}

Snippet from ArgumentParser's ParsableArguments.swift

As a reminder, validate() is here for us to make sure that the parsed arguments are valid, not type-wise, but logic-wise.

The Static Main

When following the ArgumentParser instructions, the final step is to call .main() in our type. This static method is not required from the ParsableCommand (nor from ParsableArguments etc) and we haven't defined it, it turns out that it's implemented as a public extension of ParsableCommand:

extension ParsableCommand {
  public static func main(_ arguments: [String]? = nil) -> Never {
    do {
      let command = try parseAsRoot(arguments)
      try command.run()
      exit()
    } catch {
      exit(withError: error)
    }
  }
}

Snippet from ArgumentParser's ParsableCommand.swift

The method does two things:

1. parse and initialize our command via a parseAsRoot(_:) function.
2. run it

All the magic happens in the first step, step 2 executes our run() implementation (or the default implementation we've seen above).

parseAsRoot

The purpose of this method is solely to return an instance of ParsableCommand, more specifically an instance of our type (conforming to ParsableCommand):

public static func parseAsRoot(
  _ arguments: [String]? = nil
) throws -> ParsableCommand {
  var parser = CommandParser(self)
  let arguments = arguments ?? Array(CommandLine.arguments.dropFirst())
  var result = try parser.parse(arguments: arguments).get()
  do {
    try result.validate()
  } catch {
    throw CommandError(
      commandStack: parser.commandStack,
      parserError: ParserError.userValidationError(error))
  }
  return result
}

Snippet from ArgumentParser's ParsableCommand.swift

We first obtain the input arguments via CommandLine.arguments, and then we parse them via a new CommandParser entity:
if the parse is successful, we validate the command line inputs (with ParsableArguments's validate()) and then return the new instance (of our ParsableCommand type), ready to run.

CommandParser

CommandParser structures our command as a tree: a command can contain zero or more subcommands, which can contain subcommands, which can contain subcommands, which... there's no limit set by ArgumentParser.

struct CommandParser {
  let commandTree: Tree<ParsableCommand.Type>
  var currentNode: Tree<ParsableCommand.Type>
  var parsedValues: [(type: ParsableCommand.Type, decodedResult: ParsableCommand)] = []
  
  var commandStack: [ParsableCommand.Type] {
    let result = parsedValues.map { $0.type }
    if currentNode.element == result.last {
      return result
    } else {
      return result + [currentNode.element]
    }
  }
  
  init(_ rootCommand: ParsableCommand.Type) {
    self.commandTree = Tree(root: rootCommand)
    self.currentNode = commandTree
    
    // A command tree that has a depth greater than zero gets a `help`
    // subcommand.
    if !commandTree.isLeaf {
      commandTree.addChild(Tree(HelpCommand.self))
    }
  }
}

Snippet from ArgumentParser's CommandParser.swift

In the initializer we can see how every command line tool gets a HelpCommand.

Let's look at the parse(_:) method, called by the ParsableCommand's static main():

mutating func parse(arguments: [String]) -> Result<ParsableCommand, CommandError> {
    var split: SplitArguments
    do {
      split = try SplitArguments(arguments: arguments)
    } catch {
      ...
    }
    
    do {
      try descendingParse(&split)
      let result = try extractLastParsedValue(split)
      ...

      return .success(result)
    } catch {
      ...
    }
  }

Snippet from ArgumentParser's CommandParser.swift

parse(arguments:) can be split in three steps:

1. Create a new instance of SplitArguments, a new entity, out of the input string arguments.
2. Translate the SplitArguments instance into a ParsableCommand instance.
3. Return it.

Step 1

This step turns the input string arguments into an instance of SplitArguments:

struct SplitArguments {
  var elements: [(index: Index, element: Element)]
  var originalInput: [String]
}

Snippet from ArgumentParser's SplitArguments.swift

This step has no knowledge about our command line tool definition.

SplitArguments translates the input array of strings into something more meaningful for a command line tool input: options and values. Options are anything with a dash in the front, values are strings without a dash in front.

Here's the definition of SplitArguments's Element (with comments):

enum Element: Equatable {
  case option(ParsedArgument) // something with a dash in the front
  case value(String) // values
  case terminator // --, special character
}

Snippet from ArgumentParser's SplitArguments.swift

Where we introduce ParsedArgument:

enum ParsedArgument: Equatable, CustomStringConvertible {
  case name(Name) /// `--foo` or `-f`
  case nameWithValue(Name, String) // `--foo=bar`
}

Snippet from ArgumentParser's SplitArguments.swift

Lastly, we have the Index definition:

struct Index: Hashable, Comparable {
  var inputIndex: InputIndex
  var subIndex: SubIndex
}

Snippet from ArgumentParser's SplitArguments.swift

Each Index has an inputIndex, which is the argument index in the original input, and a subIndex: for example $ tool -af has both -a and -f at the same index, but subindex of 0 and 1 respectively.

If you'd like to see the actual parsing, please see here.

Step 2

At this point we have our complete SplitArguments instance and its time to run CommandParser's descendingParse:

internal mutating func descendingParse(_ split: inout SplitArguments) throws {
  while true {
    try parseCurrent(&split)
    
    // Look for next command in the argument list.
    if let nextCommand = consumeNextCommand(split: &split) {
      currentNode = nextCommand
      continue
    }
    
    // Look for the help flag before falling back to a default command.
    try checkForHelpFlag(split)
    
    // No command was found, so fall back to the default subcommand.
    if let defaultSubcommand = currentNode.element.configuration.defaultSubcommand {
      guard let subcommandNode = currentNode.firstChild(equalTo: defaultSubcommand) else {
        throw ParserError.invalidState
      }
      currentNode = subcommandNode
      continue
    }
    
    // No more subcommands to parse.
    return
  }
}

Snippet from ArgumentParser's CommandParser.swift

As a reminder, CommandParser thinks of our command line tool as a tree structure (where each node is a ParsableCommand type): in descendingParse we are building said tree based on the parsed SplitArguments.

We do not create the whole tree structure, but only the relevant branches based on the given SplitArguments instance.

Let's have a look at parseCurrent(_:) next:

fileprivate mutating func parseCurrent(_ split: inout SplitArguments) throws {
  // Build the argument set (i.e. information on how to parse):
  let commandArguments = ArgumentSet(currentNode.element)
  
  // Parse the arguments into a ParsedValues:
  let parsedResult = try commandArguments.lenientParse(split)
  
  let values: ParsedValues
  switch parsedResult {
  case .success(let v):
    values = v
  case .partial(let v, let e):
    values = v
    if currentNode.isLeaf {
      throw e
    }
  }
  
  // Decode the values from ParsedValues into the ParsableCommand:
  let decoder = ArgumentDecoder(values: values, previouslyParsedValues: parsedValues)
  var decodedResult: ParsableCommand
  do {
    decodedResult = try currentNode.element.init(from: decoder)
  } catch {
    ...
  }
  
  // Decoding was successful, so remove the arguments that were used
  // by the decoder.
  split.removeAll(in: decoder.usedOrigins)
  
  // Save this decoded result to add to the next command.
  parsedValues.append((currentNode.element, decodedResult))
}

Snippet from ArgumentParser's CommandParser.swift

First we create a new ArgumentSet instance, which is based on the ParsableCommand type associated with the current node in the tree (if this is the first iteration, we're still at the root):

struct ArgumentSet {
  var content: Content
  var kind: Kind // Used to generate help text.
}

Snippet from ArgumentParser's ArgumentSet.swift:

Where we define a Content:

enum Content {
  case arguments([ArgumentDefinition]) // A leaf list of arguments.
  case sets([ArgumentSet]) // A node with additional `[ArgumentSet]`
}

Snippet from ArgumentParser's ArgumentSet.swift:

ArgumentSet helps us create yet again a tree structure of our command line tool, however the emphasis here is on the command line arguments instead of the command line sub/commands.

Going back to parseCurrent(_:), here's the ArgumentSet initializer that we use:

extension ArgumentSet {
  init(_ type: ParsableArguments.Type) {
    let a: [ArgumentSet] = Mirror(reflecting: type.init())
      .children
      .compactMap { child in
        guard
          var codingKey = child.label,
          let parsed = child.value as? ArgumentSetProvider
          else { return nil }
        
        // Property wrappers have underscore-prefixed names
        codingKey = String(codingKey.first == "_" ? codingKey.dropFirst(1) : codingKey.dropFirst(0))
        
        let key = InputKey(rawValue: codingKey)
        return parsed.argumentSet(for: key)
    }
    self.init(additive: a)
  }
}

Snippet from ArgumentParser's ParsableArguments.swift

This initializer is where a second magic trick happens: 1. we first initialize our ParsableArguments type with ParsableArguments's required init() 2. just to get the instance Mirror representation 3. and extract its children 4. which are the properties defined in our type, a.k.a. the ones with one of the four ArgumentParser property wrappers (@Argument, @Option, @Flag, and @OptionGroup).

This is to say, ArgumentSet initialization is where our ParsableCommand type properties auto-magically transform into parsable input arguments that can be read from the command line input.

Once we have the ArgumentSet instance, we proceed with our parseCurrent(_:) execution by matching the contents of SplitArguments (which uses as input the input arguments from the command line) with ArgumentSet (which uses as input our type definition).

This match result is then used by the ArgumentDecoder to really instantiate our command line and set all the (property wrappers) values.

The CommandParser's descendingParse(_:) continues its execution until all the arguments have been consumed:
once this is completed, we go back to CommandParser's parse(arguments:), which then extracts and returns the last parsed ParsableCommand instance. Completing the last two steps of CommandParser's parse(_:).

Wrapping Up The Static Main

At this point we are back to the ParsableCommand's parseAsRoot(_:) method, with our ParsableCommandinstance and all its properties set. There's one last step that we need to take before finally running: do our (custom and optional) ParsableArguments's input validation.

Once the validation passes, we finally run our tool, which completes the whole journey.

Conclusions

Clarity at the point of use is the first fundamental in Swift's API Design Guidelines, what the Swift team has achieved with ArgumentParser goes well beyond that: it's ease at the point of use.

The more we dig into this library the more we can appreciate how much complexity is hidden behind a protocol and four property wrappers:
with this article I hope to have given you a small glimpse into the tremendous work the Swift team put into the library, and hope you can now also appreciate how elegant this library API truly is.

Thank you for reading and please don't hesitate to let me know of any other library with such powerful and elegant API 😃

Subscribe and follow me on Twitter for more insights into Swift and all things around the language! Until next time 👋🏻