<p>Just for my own purposes, I'm trying to build a tokenizer in Java where I can define a regular grammar and have it tokenize input based on that. The StringTokenizer class is deprecated, and I've found a couple functions in Scanner that hint towards what I want to do, but no luck yet. Anyone know a good way of going about this?</p>

If I understand your question well then here are two example methods to tokenize a string. You do not even need the Scanner class, only if you want to pre-cast the tokens, or iterate through them more sofistically than using an array. If an array is enough just use String.split() as given below.
Please give more requirements to enable more precise answers.
```
```
import java.util.Scanner;

  public class Main {    

    public static void main(String[] args) {

        String textToTokenize = "This is a text that will be tokenized. I will use 1-2 methods.";
        Scanner scanner = new Scanner(textToTokenize);
        scanner.useDelimiter("i.");
        while (scanner.hasNext()){
            System.out.println(scanner.next());
        }

        System.out.println(" **************** ");
        String[] sSplit = textToTokenize.split("i.");

        for (String token: sSplit){
            System.out.println(token);
        }
    }

}
```
```