Contents

Automated Frame Counting Method

Introduction

The Automated Frame Counting method was introduced in Measuring Latency in Virtual Environments. This method measures latency by observing a system with a high speed camera. It tracks target positions and identifies the individual frames where the targets change direction. The number of frames between these samples, for one target to the other, is the latency between them.

Frame Counting has been used many times before. On first appearances counting frames would be easy, but identifying where motion begins or ends, especially with very high speed video, is difficult. The Automated Frame Counting method implements an algorithm that observes the movement of a target over a large amount of time, identifying the very first frame that the motion began, with higher speed and accuracy than a human.

The high speed camera used in our investigation was a Casio Exilim EX-ZR1000 capable of 1000fps.

Manual

  1. Begin by selecting a viewpoint in the virtual world to measure. This should contain a virtual object which moves freely in space with a tracker. In the example below we create a white box in a Unity scene. The mouse (tracker) moves the camera, so the box appears to move with it.
  2. Attach a marker to the tracker which will be visible to the camera. At 1000fps the exposure time of each frame is very low. This makes illumination a concern. We used glossy white targets illuminated with an LED lamp. Using an LED lamp prevents the AC power source modulating the illumination. LEDs or keyring torches may also make good targets.
  3. Place the camera so that it observes both the virtual and real targets within the frame.
    4. High speed cameras compromise resolution to remain cost effective. High frame rates inherently mean low exposure times. As a result taking captures for frame counting can require complex set-ups.
  4. Start the recording and move the tracker back and forth in a sinusoidal motion. The longer the capture the more reliable the result, but the longer it will take to process. In our investigation 3-5 second captures were used.
  5. Connect the camera to a computer.
  6. Navigate Matlab to the directory containing the source for the Automated Frame Counting Method (available below), and begin by running the command estimate_latency_framecounting.
  7. The program will prompt for a video file to read in, provide it the capture from the camera. Run 'help estimate_latency_framecounting' to see other ways to use the program.
  8. The program will ask if you wish to crop the video. For long captures cropping the video so that only the segments of the frames containing the trackers are processed can save considerable processing time.
    • If you do wish to crop the video, click 'Yes' and do so by dragging a box around a region containing both targets. Adjust it with the handles if necessary, and when ready double click within it to continue.
  9. After cropping has been performed, or the dialog dismissed, it will ask you to select a threshold. This is used to group each pixel into one of two clusters based on its luminance. Selecting a high value reduces the influence of noise, but risks causing the size of the targets to fluctuate as pixels on the edges move above and below the threshold. For dim captures, too low a value can cause the target to be lost entirely. Use the left and right arrow keys to select a threshold at which the targets are clearly visible. It does not matter if the silhouette of other objects are visible, so long as they will not intersect with the targets. In the command window the value of the current threshold is shown. When done close the window to continue.
    10. Ideally only the target silhouettes should be visible. This is one advantage of a low exposure time - there will be high contrast between well and poorly illuminated objects (and the only well illuminated objects will be deliberately so).
  10. The window will re-appear with the selected threshold. Nominate the real and virtual objects by clicking on them. A blue circle will indicate what objects are being tracked. When both are selected, close the window. If the frame is very small, the window can be expanded by dragging one of the corners.
    11. On the first frame the click positions will snap to the centre of the blobs so they don't have to be perfectly placed
  11. The program will now track the centroids of the selected objects through the sequence, and present the results asking if they are moving in the same direction. This window should also reveal any issues with raw tracked data. For example, if one object was lost and the tracking algorithm began to follow the other, the two waveforms will suddenly become homogeneous halfway through the capture.
  12. The scale selection window follows. The algorithm will make its best effort to guess the scale for the features, however this should always be manually confirmed. Use the arrow keys to increase or decrease the size of scale used to search for features. This size defines the width of the window used to calculate the gradient, and assess the strength of the features. The acceleration is plotted in green. The red lines indicate stopping, starting and turning points of the targets. It is the positions indicated by these lines which shall be subtracted from eachother to estimate the latency. These lines should line up with the obvious motion features in the position plots (blue), there should be only one for each feature, and the same number of features in both plots.
    13. Use the arrow keys to adjust the scale. The effect can be seen on both the recovered acceleration (green plot) and the detected features/edges (red lines)
  13. Once this window has been closed a variable containing the results will be returned. This cell array contains metrics about the latency estimation, the frame-to-frame latencies that go into this assessment, and the actual frames at which motion features were determined to occur.
    • You can use celldisp(ans) to see the content of all the result cells in the command window.

Downloads

The source code for the Automated Frame Counting Method is available here along with example captures from our investigation.