You could also try:
With Application.CommandBars.Add(Name:="My Temporary Toolbar", Temporary:=True)
With .Controls.Add(Type:=msoControlButton, Temporary:=True)
.Caption = "My Temporary Button"
.OnAction = "MyOnActionProperty"
In other words, create the toolbar/button(s) using the Temporary parameter.
You could place this code in a global XLA (Excel Addin) file to be copied into your XLSTART folder.
You'd probably then want to launch this code from something like an AutoExec procedure within the global addin.