Cross-sections of the universe at different times. If the universe is expanding today, then it was closer together in the past. If you go back far enough then it was in an extremely dense and hot state, where no normal matter could have existed.
If the distance between galaxies is increasing today, then everything must have been closer together in the past.
If we rewind back time far enough, then at some point everything was so close together that it was too hot and dense for normal matter to form. It was from this singular state that the universe has subsquently expanded.
This is the essence of the Big Bang theory: that the universe was initially in a very hot, dense state, from which it expanded and cooled, over billions of years, into the universe we see today. It is even possible to put a date on the beginning: the universe as we know it came into being 13.7 billion years ago.
This is the sense in which the universe is finite: the universe has only existed for a finite amount of time. Space might still be infinite, but because the light has not had time to reach us from the most distant reaches of the universe, it is impossible to tell the shape and size of the universe.
The expansion of the universe happened simultaneously everywhere. It didn't have a centre, nor did it explode into some external empty space. All of matter, energy and space erupted out together.
Beside the recession of the galaxies, there are four main other pieces of evidence for the Big Bang:
The age is taken from [Menegoni et al. 2009].
The age of the universe is 13.7 billion years. The most distant objects we can see are 45.7 billion light-years away. We can't see any further because the light hasn't had time to reach us yet. Thus it is hard to say how spatially big the universe really is (or whether indeed it is infinite in size), whether it is flat, or whether it has an interesting shape. It could, for example, be like a closed ribbon, where it is possible that if you go far enough in a single direction that you eventually come back to where you started.
The most popular assumption is that space is flat and infinite in extent.
[Menegoni et al. 2009]: Menegoni, E. et al, "New constraints on variations of the fine structure constant from CMB anisotropies" in Physical Review D 80 (8): 087302. arXiv:0909.3584.