I've just hit caps lock accidentally. This key is so useless; I never ever wanted to hit it.
So I deactivated it:
$ setxkbmap -option caps:none
To run this every time at startup, I've added it to /etc/rc.local.
However, it did not work. I guess it is executed too soon.
An alternative which worked is adding a file ~/.config/autostart/shift.desktop with the
following content:
[Desktop Entry]
Type=Application
Name=shift
Exec=setxkbmap -option caps:none
X-GNOME-Autostart-enabled=true
No more issues with caps lock ♡.
More functionality
I've just received a remark by Micha that one could add useful functionality. For example with Xmodmap or within the Ubuntu settings one can make this a compose key for special characters such as ·×⋄.