While the previous answers are helpful, I'll add a bit more of an explicit recipe for Org mode.
By default, Org mode adds a rather unhelpful hypersetup section. You can prevent this addition by adding the following line to your .emacs file:
(customize-set-value 'org-latex-with-hyperref nil)
In addition, if you want to pass any options to hyperref or url (which is loaded automically by hyperref), then you can use a command of the form (notice the escaped backslash):
(add-to-list 'org-latex-default-packages-alist "\\PassOptionsToPackage{hyphens}{url}")
This particular command directs the url package to break long URLs at the end of the page over hyphens.
Then, in the Org document itself, we can add the following lines:
#+LaTeX_HEADER: \usepackage[x11names]{xcolor}
#+LaTeX_HEADER: \hypersetup{linktoc = all, colorlinks = true, urlcolor = DodgerBlue4, citecolor = PaleGreen1, linkcolor = black}
The first line adds the xcolor package with X11 color names, and the second line specifies several options for the hyperref package, with the relevant option for this question being colorlinks = true.
See the Package Options section of the hyperref documentation, and the Colors By Name section of the xcolor documentation for more information on the available hyperref settings and X11 color names, respectively.