What Is the FreeD Protocol
How PTZ Cameras Simplify AR and Virtual Production
Virtual production looks simple on screen: a camera moves, and the virtual environment moves with it. But behind that seamless effect is a constant exchange of camera data — position, rotation, zoom, and lens information — that allows the graphics system to understand exactly how the real camera is moving.
FreeD is the protocol that makes this communication possible. It sends real-time camera tracking data from a camera system to virtual production platforms, allowing rendered environments to stay perfectly aligned with the camera’s perspective.
Originally created for broadcast applications, FreeD has become a widely used solution in AR, XR, and virtual production workflows. In this guide, we’ll explain what FreeD is, what data it actually carries, how it works with PTZ cameras, and why native FreeD support is making advanced virtual production workflows easier to deploy.
The Problem FreeD Solves:
Why Virtual Backgrounds Need Camera Data
To understand why FreeD exists, it helps to start with what happens when it doesn't.
The Static Green Screen Problem
The simplest form of virtual background is a static composite: a camera films a presenter in front of a green screen, the green is removed in post-production or in real time, and a background image is placed behind them. This works — but only as long as the camera doesn't move.
The moment the camera pans, tilts, or zooms, the illusion breaks. The background stays fixed while the camera's perspective shifts, and the result is immediately visible: the presenter appears to float against a backdrop that isn't responding to the camera's movement. The virtual environment has no idea the camera moved, because nobody told it.
This is not a rendering problem. The graphics engine is perfectly capable of producing a photorealistic virtual environment from any camera angle. The problem is information: the engine doesn't know what angle the camera is at, because that data isn't being delivered to it.
What the Rendering Engine Actually Needs
For a virtual background to move correctly with a camera, the rendering engine needs to know — continuously, in real time, frame by frame — the answer to four questions:
Where is the camera pointing?
Pan and tilt angles tell the engine the horizontal and vertical direction of the lens.
How far is the camera zoomed in?
The current focal length determines the field of view the engine needs to render.
Where is the focus set?
Focus distance affects depth of field rendering in the virtual scene.
Is the camera itself moving through space?
For fixed PTZ cameras, this is constant — but the engine still needs to know the camera's fixed position as a reference point.
Without accurate, real-time answers to all of these questions, the virtual background cannot match the camera's perspective. Any mismatch — even a fraction of a degree in pan angle, or a small error in zoom — produces a visible misalignment between the real foreground and the virtual background.
The Traditional Solution and Its Cost
Before FreeD became widely supported in PTZ cameras, integrating a camera into a virtual production system required external tracking hardware: mechanical encoders attached to the camera head, optical tracking markers placed in the studio, or dedicated robotic systems with built-in position sensors. These systems could provide the data the rendering engine needed — but they added significant cost, required specialized installation and calibration, and were not practical for smaller productions or venues without dedicated technical crews.
FreeD, implemented directly in the camera, changes this by reading the same position and lens data from the camera's own internal motors and encoders — the same hardware that already controls pan, tilt, zoom, and focus — and transmitting it directly to the rendering engine over a standard network connection. No external sensors. No additional calibration hardware. The data the engine needs is already inside the camera. FreeD is simply the protocol that delivers it.
What Is FreeD?
FreeD — short for Free Dimensional Video — is an open camera tracking protocol originally developed by the BBC that transmits real-time camera position, orientation, and lens data to virtual production rendering systems via UDP network packets.
Despite being developed in the 1990s, FreeD remains one of the most widely supported camera tracking protocols in the broadcast and virtual production industry. Its longevity comes from a combination of factors: it is open and royalty-free, it is lightweight enough to transmit with negligible latency, and it is natively supported by virtually every major virtual production platform in use today.
What FreeD Transmits
FreeD covers eight axes of camera data in a single transmission:
Position data (X, Y, Z): The camera's location in three-dimensional space — how far left or right, forward or backward, and up or down the camera is from a fixed reference point. For PTZ cameras mounted in a fixed position, these values remain constant, but they must still be defined so the rendering engine knows where in the virtual space the camera is anchored.
Rotation data (Pan, Tilt, Roll): The camera's orientation — which direction it is pointing horizontally (pan), vertically (tilt), and whether it is rotated around its own axis (roll). These values update continuously as the PTZ camera moves, and they are the primary data the rendering engine uses to update the virtual background's perspective in real time.
Lens data (Zoom, Focus): The current focal length and focus distance of the lens. Zoom determines the field of view the rendering engine needs to match. Focus distance affects depth of field rendering in the virtual scene, ensuring that the virtual background's blur characteristics match the physical lens.
Position
Rotation
Lens
How FreeD Transmits It
FreeD uses the D1 protocol format — a standardized 29-byte UDP packet structure. Each packet contains all eight axes of data encoded in a fixed binary format, transmitted continuously at the camera's output frame rate. The 29-byte packet size is deliberately minimal: it keeps network overhead negligible and transmission latency as low as possible, which is essential when the data needs to arrive at the rendering engine in sync with the video frame it corresponds to.
FreeD data can be delivered over three transmission paths:
IP network (UDP)
The most common method in modern deployments — the camera sends FreeD packets directly over ethernet to the rendering system
RS-422 / RS-232 serial
The original transmission method, still used in legacy broadcast facilities and some specialized installations
NDI|HX
Supported in some implementations where the video and tracking data travel over the same IP infrastructure
Which Systems Support FreeD
On the rendering side, FreeD is natively supported by the major virtual production platforms:
• Unreal Engine via the Live Link FreeD plugin
• disguise (d3)
• Vizrt / Viz Engine
• Zero Density Reality Engine
• Pixotope
• Brainstorm Infinity Set
On the camera side, FreeD support requires the camera to read its own motor and encoder data and package it into the FreeD format for transmission. This capability is built into professional broadcast PTZ cameras that explicitly list FreeD support in their specifications — including Telycam's Explore SE, which outputs native FreeD data directly to compatible rendering systems without requiring external tracking hardware or middleware.
What Is PTZF Data?
PTZF is the shorthand for the four lens and motion parameters that a FreeD-enabled camera transmits continuously to a rendering engine: Pan, Tilt, Zoom, and Focus. Together, these four values give the rendering system a complete description of what the camera is doing at any given moment — which direction it's pointing, how far it's zoomed in, and where the optical focus is set.
Understanding what each parameter actually represents — and why all four are necessary — makes it clearer why a rendering engine can't produce a correct virtual composite without them.
Pan: Horizontal Direction
Pan is the camera's horizontal rotation — how far left or right the camera has turned from its center position, measured in degrees. When a PTZ camera pans to follow a presenter moving across a stage, the pan value changes continuously. The rendering engine uses this value to update the virtual background's horizontal perspective in real time, so the virtual environment shifts in the correct direction and at the correct rate as the camera turns.
Without accurate pan data, the virtual background would either stay fixed while the foreground rotates, or move in the wrong direction — both immediately visible to any viewer.
Tilt: Vertical Direction
Tilt is the camera's vertical rotation — how far up or down the camera is angled from its center position. A camera tilting up to follow a presenter standing from a seated position changes its tilt value, and the rendering engine adjusts the vertical perspective of the virtual scene accordingly.
Pan and tilt together define the direction the camera is pointing in three-dimensional space. They are the two parameters the rendering engine updates most frequently, because most camera movement in a live production involves some combination of horizontal and vertical rotation.
Zoom: Field of View
Zoom transmits the camera's current focal length — effectively, how far the lens is zoomed in at any given moment. This value determines the field of view the rendering engine needs to match. A wide zoom produces a broad virtual background. A tight zoom produces a narrower, more magnified virtual scene that corresponds to what a telephoto lens would see.
This is why zoom data is critical even when the camera isn't physically moving. A camera can be completely stationary — no pan, no tilt — and a zoom change alone will break the composite if the rendering engine isn't told about it. The virtual background's perspective needs to match the lens's current field of view, not just the camera's direction.
Focus: Depth of Field
Focus transmits the camera's current focus distance — how far away from the lens the point of sharpest focus is set. The rendering engine uses this to match the depth of field characteristics of the virtual scene to the physical lens. When the camera focuses on a presenter close to the lens, the virtual background should appear appropriately blurred. When focus is pulled to a distant subject, the virtual environment's depth of field adjusts to match.
Of the four PTZF parameters, focus is the most subtle in its effect — but it's the one that makes the difference between a composite that looks real and one that looks like a clean technical exercise. Mismatched depth of field between the foreground and the virtual background is one of the most common tells that a scene is composited rather than real.
Why All Four Are Necessary
Each PTZF parameter answers a different question the rendering engine needs to resolve before it can produce a correct frame:
Pan and Tilt → Where is the camera pointing?
Zoom → What field of view is the lens producing?
Focus → What depth of field should the virtual scene display?
Remove any one of these, and the composite breaks in a specific, identifiable way. Remove pan and tilt data, and the background doesn't follow the camera. Remove zoom data, and the virtual background's perspective doesn't match the lens's field of view. Remove focus data, and the depth of field in the virtual scene doesn't match the physical lens — the composite looks clean but artificial.
Think of PTZF data as the rendering engine's instruction set — updated every frame, telling it exactly how to render the virtual world so it appears to exist in the same physical space as the real camera and the real presenter in front of it.
How FreeD Works in a Virtual Production Pipeline
Understanding FreeD as an isolated protocol is one thing. Seeing how it fits into a complete virtual production system — where video, tracking data, rendering, and synchronization all need to work together — is what makes it practically useful.
The Complete Signal Flow
A FreeD-based virtual production pipeline involves four parallel processes happening simultaneously, every frame:
01
Video Capture
The PTZ camera films the presenter in front of the green screen — or in front of an LED volume — producing a live video signal that travels to the compositing system via SDI, HDMI, or NDI.
02
Data Transmission
Simultaneously, the camera reads its own internal motor encoders — pan position, tilt position, zoom level, focus distance — packages that data into a 29-byte FreeD UDP packet, and transmits it over the network to the rendering engine. This happens continuously, at the camera's output frame rate, independent of the video signal.
03
Scene Rendering
Unreal Engine — or whichever rendering platform is in use — receives the FreeD packets via Live Link and uses the PTZF values to calculate the correct camera perspective for the virtual scene. Every frame, it renders the virtual background from exactly the angle, zoom level, and focus distance the real camera is currently using.
04
Compositing
The live camera feed and the rendered virtual background are combined in real time. In a green screen setup, the green background is keyed out, overlaying the presenter seamlessly onto the rendered virtual environment. In an LED wall (XR) setup, the rendering engine outputs the correct perspective directly to the LED screens behind the presenter while compositing outer-frustum background elements. The final output is a unified frame that appears to exist in a single, coherent physical space.
The Synchronization Problem: Why Genlock and Tracking Delay Both Matter
Having FreeD data and a video signal arriving at the rendering system is necessary — but it isn't sufficient. For the composite to look correct, the two streams need to be aligned in time: the FreeD packet describing the camera's position at a given moment needs to correspond to the video frame captured at that exact same moment.
This is a two-layer synchronization problem.
The first layer is frame-level synchronization — solved by Genlock.
Genlock locks the camera's video output to a shared timing reference, ensuring that frames are captured at the same moment across all devices in the system. Without Genlock, the video signal and the rendering engine's output can drift out of phase, causing the virtual background to appear slightly ahead or behind the real camera movement. For a more detailed explanation of how Genlock works in virtual production, see our guide on camera synchronization.
The second layer is pipeline latency — managed through Tracking Delay.
Even with Genlock in place, the video signal and the FreeD data don't necessarily arrive at the compositing system at the same time. The video signal passes through a capture card before entering the rendering pipeline, which introduces a small but measurable processing delay — typically equivalent to one to four frames depending on the capture hardware and system configuration. The FreeD UDP packets, being lightweight 29-byte transmissions over a direct network connection, often arrive faster than the video.
The result is a timing offset: the rendering engine has FreeD data for the current camera position, but the video frame it needs to composite it against hasn't arrived yet. In Unreal Engine's Live Link settings, this is addressed through a Delay Frames parameter — typically set between one and four frames — that holds the FreeD data until the corresponding video frame arrives, ensuring both streams are composited against each other at the correct moment.
Getting this value right is a practical calibration step in any FreeD deployment. The standard method is to place a physical reference object in the scene alongside a matching virtual object, perform a camera pan, and adjust the delay value until the two objects stay locked together through the movement.
PTZ Cameras and FreeD:
Why the Combination Makes Sense
FreeD itself is not new — the real change is where the tracking data comes from.
In traditional virtual production workflows, cameras needed external tracking systems to provide position and lens information. These systems measured camera movement and converted it into data that the rendering engine could understand.
PTZ cameras take a different approach. Because their pan, tilt, zoom, and focus movements are controlled by internal motors with built-in encoders, the camera already knows exactly where it is pointing and how the lens is changing. FreeD simply provides a standardized way to send that information to the virtual production system.
In other words, the tracking capability is built into the camera itself — no additional tracking hardware is required for fixed PTZ installations.
Why This Matters for Virtual Production
A traditional camera setup often requires additional tracking hardware, calibration, and studio preparation before it can work with virtual environments. This increases both cost and operational complexity.
With a PTZ camera that supports native FreeD output, the workflow becomes much simpler:
The camera provides real-time pan, tilt, zoom, and focus information directly to systems such as Unreal Engine, disguise, or Vizrt, allowing virtual backgrounds and graphics to follow camera movements accurately.
For fixed-position virtual studios, newsrooms, corporate studios, and education environments, this removes one of the biggest barriers to adopting virtual production technology.
Understanding the Limitation:
3DOF vs. 6DOF
There is one important distinction to understand: FreeD output from a PTZ camera does not mean the camera is fully tracked in 3D space.
A PTZ camera can provide:
Pan
Horizontal rotation
Tilt
Vertical rotation
Zoom
Lens magnification
Focus
Lens focus position
These are often referred to as rotational and lens data.
However, a PTZ camera mounted in a fixed location does not know if the entire camera body has moved forward, backward, left, or right. Those X, Y, and Z position changes require additional tracking systems.
Fixed PTZ Virtual Studio
Native FreeD is usually enough because the camera position stays constant.
Moving Camera Setup
Additional 6DOF tracking is still required because the camera itself changes position in space.
A Simpler Path to Virtual Production
For many virtual production applications, the biggest challenge is not the rendering software — it is making the camera and the virtual environment communicate accurately.
PTZ cameras with native FreeD support simplify that connection by combining camera control and tracking data in one device.
Telycam’s Explore and Explore SE is designed for this type of workflow, providing native FreeD output along with broadcast-quality imaging and long-range zoom capability. For fixed virtual production setups, it removes the need for additional tracking hardware while providing the camera data required by modern AR and XR systems.
Real-World Applications of FreeD and PTZ Virtual Production
FreeD becomes valuable anywhere a camera needs to interact with a virtual environment. By sending real-time pan, tilt, zoom, and focus data to a graphics engine, PTZ cameras with FreeD support allow digital elements to move naturally with the camera — creating a more realistic AR and virtual production experience.
Virtual Studios and Broadcast Production
Virtual studios are one of the most common applications for FreeD. Instead of building multiple physical sets, broadcasters can create different environments digitally and switch between them instantly.
With FreeD-enabled PTZ cameras, camera movements are synchronized with the rendered environment. When the camera pans, tilts, or zooms, the virtual background updates from the correct perspective, maintaining the illusion of a real physical space.
This approach is especially useful for news studios, corporate broadcasts, and other productions that need flexible sets without the cost and limitations of physical construction.
AR Graphics for Sports and Live Events
FreeD is also widely used for augmented reality graphics, where virtual elements need to stay locked to real-world locations.
Examples include:
The rendering system uses FreeD camera data to understand the camera's movement and position the graphics correctly. As the camera changes angle or zooms in, the virtual elements remain anchored instead of appearing to float on the screen.
Corporate, Education, and Streaming Applications
One of the biggest changes brought by PTZ cameras with native FreeD support is accessibility.Previously, virtual production often required expensive tracking systems and specialized studio setups. Today, smaller organizations can create professional virtual environments using PTZ cameras, software rendering tools, and existing production infrastructure.
This makes FreeD-enabled workflows practical for:
- Corporate presentations
- Online education
- Training environments
- Houses of worship
- Live streaming productions
For these applications, the goal is not necessarily Hollywood-level visual effects — it is creating a flexible production environment that looks professional without adding unnecessary complexity.
Telycam's Explore and Explore SE PTZ camera is designed for these workflows, offering native FreeD output alongside broadcast-quality imaging and long-range zoom capability. This allows fixed-position virtual production setups to integrate camera tracking data without requiring additional external tracking hardware.
FAQ
Q: Does FreeD tracking require dedicated network bandwidth when sent over IP?
A: No, FreeD data packets are extremely lightweight (29-byte UDP packets per frame) and consume negligible bandwidth. However, network stability is critical. To avoid latency spikes or packet loss caused by high-bitrate NDI or uncompressed video streams, we recommend placing FreeD traffic on a dedicated VLAN or gigabit switch.
Q: Why does my camera movement look out of sync with the virtual scene even with Genlock enabled?
A: Genlock syncs frame frequency, but video capture cards typically introduce a 1-to-4 frame processing delay that lightweight FreeD UDP packets bypass. To align them, set the Tracking Delay (Delay Frames) parameter in your rendering engine (e.g., Unreal Engine Live Link) to hold the tracking data until the matching video frame arrives.
Q: Can I use FreeD PTZ cameras with switching software that doesn't natively support FreeD?
A: Yes. If your primary switching software (like OBS or vMix) lacks native FreeD support, you can route the FreeD data into a virtual engine (such as Unreal Engine, Vizrt, or a lightweight AR middleware) to render and composite the scene first, then feed the final keyed video stream into your switcher.
Q: Will ceiling-mounted or inverted PTZ cameras distort FreeD coordinate data?
A: Not at all. PTZ camera orientation changes do not disrupt spatial tracking. You can either invert the camera’s internal pan/tilt control settings or map and invert specific FreeD axes directly within Unreal Engine’s Live Link plugin to match your physical installation setup.
Q: Since PTZ cameras offer 3DOF tracking, when should a studio upgrade to a 6DOF tracking system?
A: Native PTZ FreeD (3DOF + lens data) is perfectly suited for fixed-position setups like news desks, virtual studios, and corporate sets where the camera remains stationary. You only need a 6DOF tracking system (such as physical tracks, pedestals, or optical markers) if the camera body itself physically moves through 3D space.
Conclusion
Virtual production has traditionally required complex tracking systems, specialized hardware, and significant technical expertise. FreeD does not change how virtual environments are rendered — it changes how easily camera tracking data can become part of the workflow.
For PTZ cameras with native FreeD support, the tracking information is already available inside the camera. The camera’s internal encoders provide pan, tilt, zoom, and focus data, while FreeD provides the standard communication method for sending that information to virtual production systems.
This makes FreeD especially valuable for fixed-camera applications such as virtual studios, corporate production, education, and live streaming environments. Instead of adding a separate tracking system, productions can build a virtual production workflow around a camera that already provides the data the rendering engine needs.
When evaluating PTZ cameras for virtual production, FreeD support is more than a technical specification — it is a sign that the camera is ready to integrate with modern AR and virtual workflows. Cameras such as Telycam’s Explore and Explore SE combine native FreeD output with professional imaging capabilities, helping make virtual production more accessible without unnecessary system complexity.
Contact Telycam
Sales Inquiries
sales@telycam.com
Call Us
+86-755-33275155