Short answer
Because making use of a live process has been easier to implement than writing a static indexer. And because Ruby's metaprogramming abilities make almost any external indexer necessarily imprecise. With Ruby's introspection capabilities, a dynamic indexer can access very accurate information about of the set of defined methods and their source locations.
Even reusing an existing indexer like YARD comes with its challenges. But yeah, nothing's stopping us from having a pluggable indexer backend, aside from the seeming lack of developers interested in implementing that.
Static vs dynamic indexing
Historically, Emacs's language tooling took one of the two distinct approaches.
The first is using an external indexer program which scans the source files in the project (and maybe its dependencies), writes that information to a file, for Emacs to read the file later. The common drawbacks are precision, the limited amount of information that can be stored in such index, and the need to continually rebuild the index manually. Also often (though not necessarily so), Emacs tooling that integrates with such indexers doesn't pay much attention to the context, like the types of objects around point, and which methods they might respond to.
Examples of such tools are etags (comes with Emacs), ctags, ripper-tags, Global, and phpctags.
In the other direction, there are more integrated solutions like SLIME, Geiser, CIDER, Indium. They normally offer more capabilities, maintain a connection to a running project, and feature a REPL of some sort. A smaller example is the built-in python-mode which uses a Python REPL for code completion (when it's running).
There are also tools like ENSIME and CEDET which somewhat straddle the middle. But it seems to me that that direction works best for statically typed languages. It also requires a significant effort to develop.
Drawbacks
Indeed, the requirement to have the project already set up and running makes it less than ideal for new developers, and still somewhat of a bother for experienced ones.
That brings us to priorities. Robe was built with more veteran users in mind, developers who can grasp its architecture, and get their project into a runnable state without much external assistance. The main goal has been to make hard things (like understanding a complex code base) easier, rather than making easy things trivial.
Code completion, on a related note, has been more of an afterthought. It is only usable when mixed together with a "all symbols from buffers" completion source, which is definitely not ideal.
Advantages
When you do have a project up and running, having it in a REPL opens up nice exploration and debugging possibilities. REPL-driven development is a thing, even if we prefer testing more in the Ruby world.
Using Ruby introspection, we know about the methods defined by macros, such as ActiveRecord::Associations::ClassMethods#has_one. has_one :foo defines several methods, create_foo among them. Provided that the class defining this association has its database schema loaded (AR does this lazily), Robe can jump to these methods, and will get you to ActiveRecord::Associations::Builder::SingularAssociation. For a static indexer to get this right, it will have to carry predefined information (probably created manually) about has_one and other macros, and will have to handle every such gem on a case by case basis.
Easy (or easier) to implement, like already mentioned. Also easier to debug, in the same REPL buffer that the project is running in.
What if my code is broken during development?
It only really has to be valid when the project REPL is booting.
After that, you yourself choose when to ask Robe (or inf-ruby) to reload the project files. In Rails projects, you can do that for all project files together (with C-c C-k). And also you can reload individual files with C-c C-l in any Ruby project.
