So after some more tinkering with linux kernel, I got a decent, cleaned up version of the sources ready. Without further ado, get the sources here:
What still has issues is the keypad, I'll have to talk about it with Eric Miao since he made some changes to the original driver. The WiFi works, sadly, the module was faulty so I replaced it with another one. Also, sound is now fixed as well. I'll prepare some kind of linux installer soon.