$(cmd) gets the standard output of cmd¹, so for that to expand to the result, you need cmd to output it:
some_func() {
echo "$(( 3 + 5 ))"
}
[ "$(some_func)" -lt 10 ]
as others have already said. However, that means some_func is run in a subshell environment, so any modification to variables or anything else will be lost afterwards.
For instance, there's no point doing:
counter=0
incr() { echo "$((++counter))"; }
while [ "$(incr)" -le 10 ]...
as $(incr) will always expand to 1 (as counter will only be incremented in the subshell).
For a function to return a numerical result via an arithmetic evaluation, you'd need a shell with support for math functions like ksh93 or zsh. bash won't do.
In zsh:
counter=0
incr() (( ++counter ))
functions -M incr
while (( incr() <= 10 )); do
print $counter
done
In ksh93:
function .sh.math.incr i {
# ksh93 functions must take at least one argument
(( .sh.value = (counter += i) ))
}
while (( incr(1) <= 10 )); do
print "$counter"
done
Another alternative in ksh93 or recent versions of mksh is to use forms of command substitution that don't introduce subshells:
counter=0
function incr { print "$(( ++counter ))"; }
while [ "${ incr; }" -le 10 ]; do
print "$counter"
done
Or in mksh:
counter=0
incr() (( REPLY = ++counter ))
while [ "${| incr; }" -le 10 ]; do
print "$counter"
done
In any POSIX shell including bash, you can always return the value in a predefined variable ($REPLY is commonly used for that):
counter=0
incr() { REPLY=$(( counter += 1 )); }
while incr; [ "$REPLY" -le 10 ]; do
echo "$counter"
done
You'll find more details at this answer to another Q&A about mksh's valsub feature
¹ stripped of its trailing newline characters, and in the case of bash of its NUL characters, and here because you forgot the quotes, subject to split+glob