You ask about initiation of the selection by M-x. I don't know whether there is a simple way to cause use of the dialog box even when you do not use a menu or tool-bar button (typically initiated with a mouse). (But it's also not clear to me what M-x has to do with selecting a file, unless you mean M-x find-file or some such.)
Anyway, the reason the file selection box is shown is likely because of the settings of options use-file-dialog and use-dialog-box, which you can customize.
In addition, since you have GTK, you can customize options x-gtk-show-hidden-files and x-gtk-file-dialog-help-text, which affect the behavior of the file selection box.
See the Emacs manual, node Dialog Boxes.
C-h v use-file-dialog tells us:
use-file-dialog is a variable defined in C source code.
Its value is t
Documentation:
Non-nil means mouse commands use a file dialog to ask for files.
This applies to commands from menus and tool bar buttons even when
they are initiated from the keyboard. If use-dialog-box is nil,
that disables the use of a file dialog, regardless of the value of
this variable.
This variable was introduced, or its default value was changed, in
version 22.1 of Emacs.
To your second question: You can test whether Emacs is running in a window system (GUI) or in terminal mode, using predicate display-graphic-p. C-h f tells us:
display-graphic-p is a compiled Lisp function in frame.el.
(display-graphic-p &optional DISPLAY)
Return non-nil if DISPLAY is a graphic display.
Graphical displays are those which are capable of displaying several
frames and several different fonts at once. This is true for displays
that use a window system such as X, and false for text-only terminals.
DISPLAY can be a display name, a frame, or nil (meaning the selected
frame’s display).