Projects, 1 of 3

A command-line tool (words)

Build words, a tool that counts the words in files, with options and help, tests, and a release build.

What you’ll build

words, a command-line tool that reads the files it’s given and lists the most common words in them. It takes options and prints help made from the same declarations, reports files it can’t read and goes on, has tests, and ships as one executable. It uses std.os’s Args for its command line, std.io for files, and a Map to count.

The command line

A Struct describes the command line, and attributes say what each field is: @Positional for the arguments, @Option for an option that takes a value, and @Flag for one that doesn’t. @Program on the Struct names the tool. Args.Parse<Options>() reads the program’s arguments into an Options.

Import std.core
Import std.os

@Program(Name: "words", About: "Counts the words in files.", Version: "1.0.0")
Struct Options
    @Positional(Help: "The files to read", Name: "file")
    Field Files: String[]
    @Option(Help: "How many of the most common words to list", Short: "t")
    Field Top: Int = 10
    @Flag(Help: "Counts Word and word as one", Short: "i")
    Field IgnoreCase: Bool
End Struct

Local options = Args.Parse<Options>()
Print("files: {options.Files}")
Print("top {options.Top}, ignoring case: {options.IgnoreCase}")

$ fb run main.fb -- --top 3 -i notes.txt todo.txt

files: [notes.txt, todo.txt]
top 3, ignoring case: True

This ran as words --top 3 -i notes.txt todo.txt. fb run passes what follows -- to the program, so fb run main.fb -- --top 3 -i notes.txt todo.txt runs it the same way. Each option’s long name comes from its field’s, IgnoreCase as --ignore-case, and a field’s default is the option’s.

Help and mistakes

Args.Parse also handles -h and --help, printing help made from the same attributes, and -V and --version, and then ends the program. Add a --min-length option, to leave out short words, and ask for help:

Import std.core
Import std.os

@Program(Name: "words", About: "Counts the words in files.", Version: "1.0.0")
Struct Options
    @Positional(Help: "The files to read", Name: "file")
    Field Files: String[]
    @Option(Help: "How many of the most common words to list", Short: "t")
    Field Top: Int = 10
    @Option(Help: "Leaves out words shorter than this", Short: "m")
    Field MinLength: Int = 1
    @Flag(Help: "Counts Word and word as one", Short: "i")
    Field IgnoreCase: Bool
End Struct

Local options = Args.Parse<Options>()
Print("files: {options.Files}")

$ fb run main.fb -- --help

Counts the words in files.

Usage: words [options] [file]...

Arguments:
  <file>                         The files to read

Options:
  -t, --top <top>                How many of the most common words to list (default: 10)
  -m, --min-length <min-length>  Leaves out words shorter than this (default: 1)
  -i, --ignore-case              Counts Word and word as one
  -h, --help                     Prints this help
  -V, --version                  Prints the version

A value that doesn’t fit its field’s type, like --min-length x, prints an error and the usage line, and exits with code 2, before any of the program’s own code runs:

Import std.core
Import std.os

@Program(Name: "words", About: "Counts the words in files.", Version: "1.0.0")
Struct Options
    @Positional(Help: "The files to read", Name: "file")
    Field Files: String[]
    @Option(Help: "How many of the most common words to list", Short: "t")
    Field Top: Int = 10
    @Option(Help: "Leaves out words shorter than this", Short: "m")
    Field MinLength: Int = 1
    @Flag(Help: "Counts Word and word as one", Short: "i")
    Field IgnoreCase: Bool
End Struct

Local options = Args.Parse<Options>()
Print("files: {options.Files}")

$ fb run main.fb -- --min-length x notes.txt

error: --min-length: expected Int32, found "x"

Usage: words [options] [file]...

For more, try '--help'.

Counting words

File.ReadText reads a whole file as text. It throws an IOError when it can’t, so the program catches one for each file, says so on stderr, and goes on with the rest. WordsOf splits text into its runs of letters, and a Map<String, Int> counts each one.

poem.txt

The rain in the night,
the wind in the trees,
and the night in the rain.

main.fb

Import std.core
Import std.collections
Import std.io
Import std.os
Import std.text

@Program(Name: "words", About: "Counts the words in files.", Version: "1.0.0")
Struct Options
    @Positional(Help: "The files to read", Name: "file")
    Field Files: String[]
    @Option(Help: "How many of the most common words to list", Short: "t")
    Field Top: Int = 10
    @Option(Help: "Leaves out words shorter than this", Short: "m")
    Field MinLength: Int = 1
    @Flag(Help: "Counts Word and word as one", Short: "i")
    Field IgnoreCase: Bool
End Struct

Function WordsOf(text: String): String[]
    Local words: String[] = []
    Local current = StringBuilder()
    For c In text.Chars
        If c.IsLetter()
            current.Append(c)
        ElseIf current.Length > 0
            words.Add(current.ToString())
            current.Clear()
        End If
    End For
    If current.Length > 0 Then words.Add(current.ToString())
    Return words
End Function

Local options = Args.Parse<Options>()
Local counts: Map<String, Int> = {}
Local total = 0

For path In options.Files
    Try
        For word In WordsOf(Try File.ReadText(path))
            If word.Length < options.MinLength Then Continue
            Local key = options.IgnoreCase ? word.ToLower() : word
            counts[key] = (counts.Get(key) ?? 0) + 1
            total += 1
        End For
    Catch e As IOError
        PrintError("words: can't read {path}: {e.Kind}")
    End Try
End For

Local common = counts.Keys()
common.SortByDescending(Function(word: String) => counts[word])
Print("{total} words, {counts.Count} different")
For word In common.View(0, Min(options.Top, common.Count))
    Print("{counts[word]:>4}  {word}")
End For

$ fb run main.fb -- --top 3 -i poem.txt missing.txt

16 words, 7 different
   6  the
   3  in
   2  rain
words: can't read missing.txt: NOT_FOUND

The output shows what went to stderr after what went to stdout. counts.Keys() gives the words in the order they were first counted, and SortByDescending keeps words with the same count in that order, so the list is the same on every run. {counts[word]:>4} right-aligns the count in four columns.

A package with tests

As the tool grows, its counting moves into a module of its own, where tests can reach it. fb new words makes the package. counting.fb holds WordsOf and a Tally Class, main.fb the command line, and tests/ the tests, which use the package through its Public names.

project.toml

[project]
name = "words"
version = "1.0.0"

counting.fb

Import std.core
Import std.collections
Import std.text

/// The words of `text`: its runs of letters, in order.
Public Function WordsOf(text: String): String[]
    Local words: String[] = []
    Local current = StringBuilder()
    For c In text.Chars
        If c.IsLetter()
            current.Append(c)
        ElseIf current.Length > 0
            words.Add(current.ToString())
            current.Clear()
        End If
    End For
    If current.Length > 0 Then words.Add(current.ToString())
    Return words
End Function

/// How many times each word comes up, in the order each first came up.
Public Class Tally
    Field counts: Map<String, Int>
    Field ignoreCase: Bool
    Field minLength: Int
    Field total: Int

    Public New(ignoreCase: Bool, minLength: Int)
        counts = {}
        Self.ignoreCase = ignoreCase
        Self.minLength = minLength
    End New

    /// Counts each word of `text`.
    Public Method Add(text: String)
        For word In WordsOf(text)
            If word.Length < minLength Then Continue
            Local key = ignoreCase ? word.ToLower() : word
            counts[key] = (counts.Get(key) ?? 0) + 1
            total += 1
        End For
    End Method

    /// How many words it has counted.
    Public Property Total: Int
        Get => total
    End Property

    /// How many different words it has counted.
    Public Property Distinct: Int
        Get => counts.Count
    End Property

    /// The `count` most common words, with how many times each came up, most first.
    Public Method Top(count: Int): (String, Int)[]
        Local words = counts.Keys()
        words.SortByDescending(Function(word: String) => counts[word])
        Local top: (String, Int)[] = []
        For word In words.View(0, Min(count, words.Count))
            top.Add((word, counts[word]))
        End For
        Return top
    End Method
End Class

main.fb

Import std.core
Import std.io
Import std.os
Import words.counting

@Program(Name: "words", About: "Counts the words in files.", Version: "1.0.0")
Struct Options
    @Positional(Help: "The files to read", Name: "file")
    Field Files: String[]
    @Option(Help: "How many of the most common words to list", Short: "t")
    Field Top: Int = 10
    @Option(Help: "Leaves out words shorter than this", Short: "m")
    Field MinLength: Int = 1
    @Flag(Help: "Counts Word and word as one", Short: "i")
    Field IgnoreCase: Bool
End Struct

Local options = Args.Parse<Options>()
Local tally = Tally(options.IgnoreCase, options.MinLength)
For path In options.Files
    Try
        tally.Add(Try File.ReadText(path))
    Catch e As IOError
        PrintError("words: can't read {path}: {e.Kind}")
    End Try
End For

Print("{tally.Total} words, {tally.Distinct} different")
For entry In tally.Top(options.Top)
    Local word, times = entry
    Print("{times:>4}  {word}")
End For

tests/counting_test.fb

Import std.core
Import std.test
Import words.counting

@Test
Function SplitsAtWhatIsntALetter()
    Expect(WordsOf("rain, wind--and night") == ["rain", "wind", "and", "night"])
    Expect(WordsOf("").Count == 0)
End Function

@Test
Function CountsEachWord()
    Local tally = Tally(False, 1)
    tally.Add("the cat and the hat")
    Expect(tally.Total == 5)
    Expect(tally.Distinct == 4)
    Expect(tally.Top(1) == [("the", 2)])
End Function

@Test
Function LeavesOutShortWords()
    Local tally = Tally(True, 4)
    tally.Add("The rain in the night")
    Expect(tally.Top(5) == [("rain", 1), ("night", 1)])
End Function

Output

     Running 3 tests
test words.tests.counting_test.CountsEachWord ... ok
test words.tests.counting_test.LeavesOutShortWords ... ok
test words.tests.counting_test.SplitsAtWhatIsntALetter ... ok

test result: ok. 3 passed; 0 failed; 0 skipped; 0 filtered out; finished in 0.02s

The output is fb test’s. Top gives (String, Int) tuples, which Local word, times = entry takes apart, and which == compares element by element in the tests.

Shipping it

fb build --release

build/words, or build\words.exe on Windows, is the whole tool: std’s code is linked in, so it needs nothing installed beside it. Copy it to a folder on your PATH, and run it anywhere:

words -t 5 -i notes.txt

What you learned

A Struct with @Positional, @Option, and @Flag describes a command line, and Args.Parse<T>() reads it, with help and errors made from the same attributes.
File.ReadText throws an IOError that the program can catch per file and go on.
A Map<String, Int> counts, and SortByDescending orders by count, keeping ties in order.
A package’s tests/ folder tests it from outside, through its Public names.
fb build --release makes one executable to ship.

Report an issue with this page or with FutureBlitz on GitHub.