The below would work. For the sake of clarity, I am using :ensure org-plus-contrib which deviates a bit from the example in your question.
(use-package org
:mode (("\\.org$" . org-mode))
:ensure org-plus-contrib
:config
(progn
;; config stuff
))
Here are some explanation notes for the above:
(use-package FEATURE-NAME
FEATURE-NAME is what goes in the (provide ..) line of a package. For org-mode, that line is (provide 'org).
:mode ((FILE-REGEXP . MAJOR-MODE))
FILE-REGEXP is the regular expression for which you want a particular major mode to be enabled. You had that correct: "\\.org$".
MAJOR-MODE is the name of the major mode that you want to enable (including the -mode part of the major mode name). In this case, the major mode name is org-mode.
- The style
:mode (FILE-REGEXP) that you used would only work if FEATURE-NAME and MAJOR-MODE are the exact same. Example: cperl-mode.
:ensure PACKAGE-NAME
PACKAGE-NAME is what I see listed in the package list (M-x package-list-packages). For clarity, in my solution above, we are installing the org-plus-contrib1 package. This is to demonstrate the feature name, major mode and the actual package name do not have to be the exact same. And also I prefer installing that instead of plain org-mode as you get all the contrib/ package goodies too! :)
1 You will need to add (add-to-list 'package-archives '("org" . "http://orgmode.org/elpa/") t) to your emacs config in order to install org-plus-contrib. See the official org-mode installation page for more info.
Note
The best way to make the above use-package snippet to work is to save it to your emacs config first, then delete prior Package Manager installed versions of org, and then restart emacs.