face tracking touch screen multitouch video game mash up?

As technology improves, actions and activities we have never before imagined become not just possible, but common. In order for this to happen though, people need to think up those weird combinations and applications for existing and emerging technologies. So, I was struck with the thought, of a fully interactive world, that just used the world we already have to create itself.

baby steps though. So the first step is finding a high quality touch screen display, with multitouch capabilities, and capturing large amounts of video with things like 360 cameras. when you take the video, have software along the vein of face tracking software map the parts of the body, and identify things that can be interacted with (such as light switches). Map a generic rig (the virtual bones and ligatures that make animation move) onto the image using the analysis of the face tracking software. If you have enough video captured you can interact with the people in the video.

If I am not explaining this well, I apologize, but I really think it has potential. It's the start of a way to make highly realistic, reactive computer programs. almost like the holodeck on star trek. Captures could include a fairly limited suite of positions and expressions. Taking angry, sad, happy, shocked, conflicted, honest, lying, and maybe exasperated, and you would have all the emotion you needed. Have postures that match, plus a few like, falling, running, jogging, depending on the nature of the features being used you could also include a few gymnastic positions (splits, back bends, high kicks, rolls, flips), and maybe some wire work (think wushu flying kung fu stuff). With that suite postures and expressions in place, you could put them in a virtual environment, and have them react and interact.

What would be left, essentially, would be collision detection, lighting, and some basic scripting. These are all things we already have the capabilities for already(though I in no way wish to downplay their difficulty or importance).