I do not know Japanese literature well but nonetheless I recommend the following:
1. Out: A Novel, by Natsuo Kirino. Vicious fun. Dark, violent, etc.
2. Kobo Abe, The Woman in the Dunes. He has been called the Japanese Stanislaw Lem. Why do I never hear about this book? The movie by the same name is good too.
4. Mishima, Spring Snow, others.
5. Haruki Murakami. My favorite is Hard-Boiled Wonderland (one of my favorite books period) and then Underground, a modern classic of social science (really). I like most of them but I feel he is repeating himself as of late.
6. Shusaku Endo, Silence. Very powerful and I remain fascinated by Japan’s so-called "Christian century."
7. Kenzaburo Oe: I like Teach us to Outgrow Our Madness.
Question: Is Tale of Genji actually fun to read? I would say about half of it, so yes it is worth the time. The best parts are very beautiful and mysterious and unlike anything else in literature. Pillow Book of Sei Shonagon is fun and is a good introduction to the period.
The bottom line: There is lots and lots more that I have never heard of, not to mention manga.