Starting with the first task, only perform the change for closed {}:
printf '%s' '{$aaa} \\{$bbb} \{$ccc} {$ddd' | sed 's/{$\([^}]*\)}/${\1}/g'
This is, everything after the {$ up to the closing } is put in \(\), so we can backreference it in the replacement string as \1. Output:
${aaa} \\${bbb} \${ccc} {$ddd
Now, about the backslash-escaping. Your solution works for \{$aaa} (don't touch) and \\{$aaa} (replace), but what about \\\{$aaa}? It will perform the change, which is probably not desired.
So I suggest to get rid of all double-backslashes first. We replace them by a character combination that cannot be part of the string, #1 as an example, and change it back in the end. This way we can concentrate on the single backslash only:
printf '%s' '{$aaa} \\{$bbb} \{$ccc} \\\{$eee} {$ddd' | sed 's/\\\\/#1/g;s/{$\([^}]*\)}/${\1}/g;s/#1/\\\\/g'
And finally we can change \{ to #2 and replace it back at the end to get it out of our way:
printf '%s' '{$aaa} \\{$bbb} \{$ccc} \\\{$eee} {$ddd' | sed 's/\\\\/#1/g;s/\\{/#2/g;s/{$\([^}]*\)}/${\1}/g;s/#2/\\{/g;s/#1/\\\\/g'
Result: ${aaa} \\${bbb} \{$ccc} \\\{$eee} {$ddd
And this will work for any number of backslashes.
But if there is no character like # you can freely use? You can use the newline, because a newline can't be part of a line. Since you seem to use GNU sed, you can do
printf '%s' '{$aaa} \\{$bbb} \{$ccc} \\\{$eee} {$ddd' | sed 's/\\\\/\n1/g;s/\\{/\n2/g;s/{$\([^}]*\)}/${\1}/g;s/\n2/\\{/g;s/\n1/\\\\/g'
If this should run on a POSIX sed, you need a workaround
{$foo} {$abc {$bar}and can there be spaces inside{$...}– Sundeep May 01 '17 at 14:11{$and}– Jun May 01 '17 at 15:38{and/or}characters inside? i.e nesting... also, what should be output for{$foo} {$abc {$bar}? – Sundeep May 01 '17 at 15:41{$...}there should only have[aZ-_]characters. This will fix the issue? – Jun May 01 '17 at 16:25