I noticed this drop down when posting a blog post, but it is empty when i click on it, even though i have created a bunch of albums.
Bellow the dropdown, it says:

You can associate an album from the Gallery with this entry to display images alongside the entry content

How does this work? It seems like i can show a gallery along with a blog post but how do i do this? Also, can it work the other way around to assign a blog post to a gallery?

 

My site bloggers use this all the time.  

You can associate an album to a blog post, which displays photos from the album at the bottom of the blog post.  You need to have the albums already created.  

No, does not work the other way around.