Did you navigate to the folder where you keep your video files in the PS3 Media Server on your computer? After selecting it may take a couple of minutes for the red X to turn to the green check.
If that does not work, close out of the program, reboot then open media server back up. If you have selected the proper folder this should do it.
The media server plays everything I throw at it. MKV, MPEG, MPEG4, AVI (even those pesky ones that won't play in WMP), and what ever else I have in my vast library. Not sure about FLV though.
Sorry, I am at work now so I can't remember exactly how to navigate to folder...I just did it the other day to when I moved all my files over to my new 2TB drive lol.