There's this open question of what Google is going to be a decade or more from now. Google X isn't the only answer to that question, but it was built as a place to do some of the exploration to find some great new problems for Google to tackle.
VisiCalc and WordPerfect were the killer apps of their day, but Google and Facebook make them look small in comparison.
Google[x]'s Focus on the Physical World
One of the missions of Google[x] is to use technology to get technology out of the way
If we want to help Google become something meaningfully different in the future, then that's more likely to happen if we focus on the physical world instead.