Now the PS3 will not play files in the MKV Container as is. So don't try putting them directly on the HDD. I'd like to share my experience with muxing and demuxing these files and WMVs for the benefit of others in the similar situation.
As an example, I will be using an MKV with x264 video and DTS 5.1 audio. Transcoding to a different codec might make this file more compatible with the PS3, but the problem is any sort of transcoding operation WILL decrease video quality. So all we're aiming to do is demux and remux, and convert the audio alone. And I'm not going to deal with subtitles either.
- MKVToolnix (MKV-extract GUI in particular)
- TsMuxer GUI
Load up your MKV file in MKV Extract GUI. The list of tracks contained in the MKV is displayed. Here select just the MPEG4/ISO/AVC and DTS tracks (check the boxes) and hit extract. The destination directory will contain the output in the form of a .264 and .dts file. The .dts file is the audio portion and I downmix that to 2.0 AC3 using WinFF. Open up WinFF and drag and drop the .dts file into it. Select Audio from the drop down menu and AC3 384kbps from the subcategory. The end result after you convert this will be a .ac3 file.
We now have all we need to make the m2ts output that will play on the PS3. Start up TsMuxer GUI. Drag and drop both the AC3 and the X264 parts into TsMuxer. There is one alternative way to do this too, without extracting the video (x264) from the mkv. TsMuxer can directly read the MKV and it will list the DTS track as well as the video track. Leave the video selected, uncheck the DTS audio and add the AC3 audio to the list of inputs. Check the box next to the AC3 track too.
Click on the track that says H.264. If the number next to "Change level" is greater than 4.1, enable the option "Change level" and choose 4.1 from the drop down menu. Else it won't play on the console. Choose "M2TS muxing" as the output option. Also, if your video is greater than 4gb, you won't be able to put it on a FAT32 formatted drive (whether it be a pen drive or an external HDD). And the PS3 cannot read NTFS formatted media.
So use the split and cut tab to choose the "Split by size" option and enter a value less than 4gb. Then go back to the Input tab and hit start muxing. Your output file will be a .m2ts file (or two or three of them if you had a large video to begin with). Copy these to the FAT32 formatted media, and enjoy HD awesomeness on your LCD or Plasma via the black box (assuming you have a HD ready or Full HD screen that is).
I'll post on how to rectify the WMV-HD video for playback on the PS3 in another post.
Till then, I hope this helps. And do note that the tools mentioned are all freely downloadable.
UPDATE: Please note that I'm downmixing 5.1 DTS to AC3 2.0 in the above example. However if you wish to convert the DTS 5.1 to AC3 5.1, just Popcorn MKV AudioConverter (links to tutorial and download), its a free utility. Then use the converted AC3 5.1 audio equipped MKV directly in TsMuxer.
The reason I'm not directly using 5.1 DTS is because the latest version of TsMuxer gives me an output with only video when I use the MKV directly with DTS. Might just be some problem with my computer and TsMuxer not getting along well but thought I'd suggest an alternate.