Yes, but exactly how depends on which shell you're using.
In case of bash, edit ~/.bashrc and add a line saying
xmodmap ~/.Xmodmap
If you don't have a ~/.bashrc yet, create one and start it with
#!/bin/bash
if [ -f /etc/bash.bashrc ] ; then
. /etc/bash.bashrc
fi
xmodmap ~/.Xmodmap
If you're using a different login shell such as tcsh or dash or zsh, you'll need to check its manpage to see what files it reads and executes on startup and add the command there. If you're unsure what login shell you're using, the SHELL environment variable should have been set, and you can check that instead with echo $SHELL .
EDIT: My bad. ~/.xsession is the correct file to add this.