I really don't know if H.264 can be used as a motion background. However, assuming you are using Encore CS3 I would do another experiment on your system.
I would go into the Encore Library and look for what I think is probably the only HD menu there (?Crib or something) and import it into your project. I believe this is 1920x1080. When you do that it brings in a motion menu which is a quicktime file .mov.
That should work and burn to a disk.
What I'm saying is that to do your project, the motion background has to be either an .avi or .mov both of which I have done. I personally have never tried H.264 since the others work for me.
Hope that helps.
John Rich