Technology Overview

Here's a more detailed description of how the system works.

  1. MultiTouchManager is constructed.
  2. MultiTouchManager constructs MultiTouchConnection, which will immediately connect to the url (default="localhost") to the cross domain policy file port (default=5503). It will then wait 2 seconds and start connecting to port 5501 and processing camera data.
  3. ...other initialization...
  4. Previous redraw cycle completes
  5. MultiTouchConnection receives camera data

  6. MultiTouchConnection searches for complete <sample>...</sample> nodes, and parses them into a Sample object. Sample has a list of new Fingers, a list of existing Fingers, and a list of no longer present Fingers. Same for new, existing and no longer present Hands.

    1. For each finger entry in XML, if an existing Finger object cannot be located by ID, create a new Finger. Mark that Finger as a new Finger in the Sample, otherwise mark it as an existing Finger. Then update the Finger position and age.
    2. For each hand entry in XML, if an existing Hand object cannot be located by ID, create a new Hand. Then update the Hand position and age, and associate Fingers with the Hand. If a Hand has non-null owner, mark the Finger as owned by the owner of the Hand.
    3. For each <raisefinger> entry the raised finger is marked as a no longer present finger in the Sample.

  7. MultiTouchConnection finishes parsing one <sample> node, and sends Sample to MultiTouchManager and/or other user built event managers

  8. MultiTouchManager, if enabled, processes the Sample.

    1. If MouseEvents are enabled, first send MouseEvents:
    2. See if we already have a CursorFinger. If CursorFinger is null, select the first new finger as the CursorFinger, and dispatch a MouseEvent.OVER and MouseEvent.DOWN to the object it hits. Otherwise, if CursorFinger is not null, see where it is. Send MouseEvent.OUT and MouseEvent.OVER events if CursorFinger moves from one object to another. Otherwise send MouseEvent.MOVE to the object the CursorFinger is on top of. If CursorFinger is in the list of expired fingers, send MouseEvent.UP and MouseEvent.OUT and set CursorFinger = null. Also send MouseEvent.CLICK if finger has not moved too much, etc.
    3. Then start sending MultiTouchEvents:
    4. For each new Finger, see if Finger hits(*) an InteractiveObject. If true, dispatch OVER event and then dispatch DOWN event to that InteractiveObject. Mark which InteractiveObject the Finger hits in this Sample, in order to dispatch OUT/UP events during processing of future Samples. Create a potential TAP event by marking down where the Finger came down. When new Finger is processed, double-check that its owner is still null. If Finger has owner, send OVER and DOWN events to owner directly, if the Finger hits the owner. Also if Finger is grabbed by an owner different from the already hit InteractiveObject, send OUT event to hit InteractiveObject.
    5. For each updated existing Finger, check if its current position or age is out of bounds of the potential TAP event. If so, delete the potential event.
    6. For each updated existing Finger with not null owner, dispatch a MOVE event directly to owner.
    7. For each updated existing Finger with null owner, see if that Finger hits an InteractiveObject. If so, check whether in the previous camera sample it hit a different InteractiveObject. Dispatch OUT event to previous InteractiveObject, if different and not null. Dispatch OVER event to new InteractiveObject, and mark that the Finger has now hit this new InteractiveObject, if different from previous sample. If new InteractiveObject is the same as the previous InteractiveObject, dispatch a MOVE event to it. If the hit InteractiveObject changes, delete any potential TAP event.
    8. For each expiring Finger, dispatch OUT event and then dispatch UP event to the Finger's previously hit InteractiveObject.
    9. For each expiring Finger, if they still have a potential TAP event, dispatch it to the previously hit InteractiveObject.
    10. After processing all Fingers in the Sample, dispatch a SAMPLE_ENDS event to all InteractiveObjects that have received events during this sample.

  9. If MultiTouchConnection has more data, repeat process until no more complete <sample> nodes are available.

  10. When MultiTouchConnection has no more data, process frame rendering normally. Cycle repeats.

*:"hit" above means a finger hitting an InteractiveObject according to the same Logic which Flash applies to see which InteractiveObject is hit by the mouse cursor. I.e. visibility, mouseEnabled and mouseChildren are checked and respected. The only difference is that objects outside the multitouch root will not ever be hit, even though they are InteractiveObjects and have mouseEnabled = true.