the books on the shelf or the books in the shelf

On means that it is on top of something (like a table).  In means that it is inside of something (like a box). 

The book is on the shelf.  The only thing in the shelf is wood.

The book is in the bookshelf.  The bookshelf is the entire piece of furniture, like a box that is open on the side.

I agree 100% with Patty, and would just add that The book is on the bookshelf would mean that the book is located on top of that piece of furniture, not on any of its shelves.  When you have as many books as I do, that is a distinct possibility.  :)