There are two problems I found with the code you wrote.
problem 1
(funcall #'(average-damp 'square) 10)
funcall requires a function as it's first argument. However, here you're trying to pass in the unevaluated list
(average-damp 'square) as if it were a function. But it's not, it's data.
Remember that #' is a shorthand for function and function does not evaluate it's arguments (see anonymous functions). So it does not evaluate (average-damp 'square).
problem 2
The other problem is your definition of average-damp.
(defun average-damp (f x)
(lambda (x) (average x (funcall f x))))
lambda does not evaluate it's body. Therefore the f that you're passing into average-damp won't end up replacing the f in (funcall f x) which it seems like is what you want.
As an illustration, this is what your version of average-damp returns when passed in with square.
(average-damp 'square) ;=> (lambda (x) (average x (funcall f x)))
Note that the f still hasn't been replaced with square.
Consequently, when the lambda this function is called, it won't know what f is (unless you defined a global variable f) and you're bound to get a Symbol's value as variable is void: f error.
solution
To address problem 2 you can use backquote to ensure that the value of average-damp's parameter, f, is replaced with square.
And to address problem 1, you should remove the #'. You don't need it because you want (average-damp 'square) to be evaluated so that it returns a function, which is what funcall requires as it's first argument.
(defun average-damp (f)
`(lambda (x) (average x (funcall #',f x))))
(funcall (average-damp 'square) 10) ;=> 55.0