Last updated:
Welcome to Lens Tweaker V3. In this document, we will be discussing how to calibrate a lens using Mo-Sys Lens Tweaker. The Lens Tweaker comes with Mo-Sys VP Pro Plugin allowing for fast lens editing and deployment within your virtual production workflow. For any questions about Mo-Sys VP Pro or about the lens calibration process, please email support.mo-sys.com.
In this section we are going to briefly explain the full Mo-Sys lens workflow. You can see the diagram in Figure 1 below.
Figure 1. Lens workflow overview
First, Mo-Sys performs a capture of the lens using the lens calibration robot. Then we process the captured data and perform an automatic calibration which then is used to create a base lens file. This file then gets uploaded to the online library. Customers can find the lens on the Mo-Sys lens website (lens.mo-sys.com) and purchase it. The lens file will get scaled to match the sensor size of the camera selected by the customer. After purchasing the lens file, the customer can perform a tweak using the lens tweaker to fine tune the calibration to their specific lens/camera setup. This manual will help you to do that.
Make sure to set the mode of Mo-Sys VP Pro Plugin to AR in Project Settings > Mo-Sys VP > Mo-Sys VP > Mode.
Within your project, click on the Mo-Sys dropdown menu and select Open Lens Tweaker as seen in Figure 2.
Figure 2. Actions and Settings
Now we need to turn our attention to the lens tweaker. You will likely be met with Figure 3 at first.
Figure 3. Lens Tweaker widget before setup
You will need to create a Live Link source to connect to your tracking solution via the Unreal _Live Link Panel_.
After you have set up your Live Link, click Load to load the lens tweaker level. Otherwise, the level can be found in the Mo-Sys VP Pro Plugin's StarterContent/Maps folder under the name L_MoSys_Tweaker_Level.umap. If you are working on your current level, be sure to save it before clicking the Load button.
Make sure your Live Link subject is set in MoSysCamera actor, MoSysLiveLink component in Subject Representation field.
Now that we have the Live Link set up and the scene loaded, all we need to do is enter PIE (Play In Editor) to start your calibration.
In this section, we will be looking at selecting and loading a lens file.
Figure 4. Lens files tab
First, you will need to select your tracking solution as seen in Figure 5.
Figure 5. Trackers list
Selecting a tracker here will automatically assign the selected tracking solution to the account that is currently logged into Mo-Sys VP Pro.
If your tracking solution does not initially appear, try hitting the refresh button to rescan for devices on your network.
As you can see in Figure 6, lens file section is made up of two parts:
Figure 6. Lens files list
Your lens files are a list of all the lens files connected to your account. You can purchase lens files at lens.mo-sys.com.
When you select a lens file, the Selected Lens History will be populated with the history of all the tweaks that you, the user, have made on a lens file. By default, if a user has no previous tweaks on a lens file, there will be a base file available for tweaking.
After selecting a lens file from the Selected Lens History, this file will automatically be downloaded and loaded, ready for tweaking. This process will take slightly longer for zoom lens files.
Figure 7. Lens information panel
After a lens file is successfully downloaded and loaded, information about this lens will automatically populate the Selected File section. In this section, you can change the serial number of the lens and camera as well as give a short and more in-depth description in the comments field. The short description will become the display name for the Selected Lens History section. Upon expanding File Details section, you will be able to see more information about the lens file (See Figure 8).
Figure 8. File details
Figure 9. Upload file buttons
These buttons are mostly used at the end of the calibration process. Here you can store your calibration in your personal collection of files, deploy your lens file to the selected tracker and use advanced options.
When a user stores a lens file, the file must have the following filled in:
Figure 10. Advanced options panel
In the Advanced Options section (Figure 10), you can do the following:
This section goes through the Scene Setup widget. It also looks at the different graphics you can use and where they will be best used.
Figure 11. Scene setup tab
Figure 12. Graphic type dropdowns
There are three different graphic types, each is best in different situation.
Poles are great for sample editing. This allows you to easily see and match your lens distortion with a real-world reference, such as a c-stand.
Cones are best for Global Zoom Tweaks. This gives you a pinpoint with a larger graphic to match up with a user's 0 point.
A cube is best for real world testing with a large graphic. The cube is $1m^3$, allowing the user to place a real-world cube in the scene and check both distortions and EPD as field of view and centre shifts.
Figure 13. Graphic placement options
The user can place a selected graphic in two ways: by distance or by coordinates.
Distance placing is most commonly used with the pole graphic. The user should input the distance from the image sensor of the camera to the position they want to place their object. In most cases, this will be the distance from the image sensor of the camera to a real-world object.
The graphic can then be placed using the place-on-crosshair button (Figure 14).
Figure 14. Place on crosshairs button
This will place the graphic on the crosshair.
Alternatively, the graphic can be placed by clicking on screen with a mouse cursor. To do that, click place-by-mouse-click button (Figure 15).
Figure 15. Place by mouse click button
For coordinate positioning, you need to select Coordinates in the following dropdown.
Figure 16. Coordinates vs Distance dropdown
This will then let the user enter an XYZ coordinate into the right most column (Figure 17).
Figure 17. Graphic Coordinates
XYZ in the Mo-Sys standard is across for X, forward for Y, and up for Z. As you change this value, the graphic position will automatically update.
Visibility button allows the user to hide and show the graphic.
Figure 18. Visibility button
To display graphic numbers, checkbox called Identify Graphics can be used (Figure 19).
Figure 19. Identify graphics button
If you have selected a pole as your graphic, you can adjust its width by changing the Scale value (Figure 20).
Figure 20. Pole scale slider
In this section we will be looking at the functionality and options in Tweak tab.
Figure 21. Lens tweaks panel
This part of the tab shows you the three methods of lens calibration. Near-Wide, Global Zoom Tweak, and Advanced Tweak (Figure 21). There is also an Edit Focal Distance panel which allows user to edit/add focal distance data. More on that in Section 4.3. Note: When tweaking a prime lens, Near-Wide method will not be available.
This method of calibration looks at a lens files split into their zoom ranges (every sample with a consistent zoom value, see Figure 22). The user can then scale the field of view and distorted field of view based by a value entered by the user in the Scale Fx field. Centre shift offset can be added using Add Cx and Add Cy fields. Finally, the user can add offset to Entrance Pupil Distance (EPD) by entering a value in EPD Add.
Figure 22. Global zoom panel
These changes will be applied across the zoom range.
Each zoom range is viewable as a collapsible dropdown, allowing the user to review all changes before saving.
By selecting Save the changes get saved into the lens file and smoothing is applied.
Figure 23. Near-wide tweak panel
Near-Wide tweak allows to simplify the tweaking process by tweaking just the far and the near ends of the lens. Linear offsets will be applied for the rest of the lens range. This usually gives good results provided the base calibration was not extremely far off. This method can only be used with zoom lenses.
To perform a tweak using a Near-Wide method, go to one end of the lens and click Add Tweak button. Then adjust the parameters until the graphics match the real world. After that, go to the other end of the lens and click Add Tweak button and adjust the parameters again. Values can be revisited and modified again by changing the lens position to that tweak point. The nearest tweak point will automatically be selected in the tweak list as you change the zoom position (Figure 24).
If parts of the lens file still look off after performing tweaks on the near and the far ends of the file, more tweaks can be added. To do that, go to the zoom position where calibration needs tweaking and click Add Tweak. Keep in mind that the first 2 tweaks always have to be the near and the far end of the lens.
Figure 24. Tweaks list
To remove a tweak, use the trash can icon on the right. To clear all tweaks and start over, click Clear Tweaks button (Seen in Figure 25).
If the user prefers to keep the current lens parameters on one end of the lens but modify the other end, then add the tweak without editing the values.
After performing the tweaks, last thing to do is to click Process button (Seen in Figure 25). This will save the tweak and apply smoothing.
Figure 25. Near-wide tweak parameters section
Near-Wide tweak mode has an option to display graphs (See Figure 26).
Figure 26. Near-wide tweak graph
To open it, click Show Graph button in the Near-Wide tweak panel. It updates real-time and allows you to see the values of the lens file and the tweaks that are about to be applied. There are two curves in the graph: the white curve represents values from the existing lens file and the coloured curve represents the new tweaked values. The red point is parameter value at current active tweak point and the grey point represents the other tweak point that is not currently being edited.
This method of calibration is for experienced users only. In this section we will be discussing the lens parameters.
Figure 27. Advanced Tweak Panel
This area consists of 8 main lens parameters plus an extra parameter for anamorphic lenses, they are:
Virtual Camera Fx is a representation of field of view measured in pixels.
Entrance Pupil Distance is the distance from the image sensor to the entrance pupil (also known as the point of no parallax) measured in metres.
Principle Point Cx and Cy are your centre shift values. This is measured as pixels.
K1, K2, and Distorted Fx are all to do with distortion values that can be changed to match observed distortions in the image.
Anamorphic Squeeze Factor is the value that defines how much an image is squeezed in anamorphic lenses. Virtual Fy and Distorted Fy cannot be changed directly and are displayed for information purposes only. They change as Anamorphic Squeeze Factor changes.
This process allows user to edit values in individual samples. When the lens position is on a sample, the sample can be edited by selecting the Edit button.
Upon beginning the edit, the edit button turns into a Store button. This is used to store changes to the current sample. If the lens is not on a sample point, editing a sample will edit the closest sample to their current position. After making changes to a given sample, the user can select Apply Selected in Range. This will push the currently selected values, indicated by the toggle tick-boxes to the right of the parameters, through the rest of the zoom range of the current sample. It is advised to use this at the beginning of each zoom range. After completing the full tweak, click Save button to save the calibration.
With the Advanced Tweak option, the user can also add extra samples. This can only be done outside of sample editing.
Figure 28. Add samples section
When adding samples, this does not add a single sample, but instead a line of samples across zoom or focus range. This can be done by entering a percentage you want to add and selecting either Add Zoom Range or Add Focus Range.
If the user adds a zoom range, it will add as many samples as there are focus points in the lens file. For example, on a standard file, if a user adds a 10% zoom range, they will make samples at (10Z, 0F), (10Z, 25F), (10Z, 50F), (10Z, 75F), and (10Z, 100F).
When adding a focus point, the result will be the same but for focus. The tweaker will add as many samples as there are zoom points in the lens file. For example, if a user adds an 85% focus sample, they will have added (00Z, 85F), (20Z, 85F), (40Z, 85F), (60Z, 85F), (80Z, 85F) and (100Z, 85F).
All tweak methods have an option to tweak Fx and CxCy by clicking and dragging on the viewport. If Fx is selected in Drag And Drop menu (Figure 28), make sure your calibration object is on the edge of the screen. Then, click on the virtual object and drag it so it matches the real world object. If CxCy is selected, make sure your calibration object is in the centre of the screen and then click on the virtual object. Drag it so it matches the real world object. Off button will disable tweak on Drag And Drop.
Figure 29. Drag and drop menu
Figure 30. Lens data panel
The Lens Data Panel is broken down into 3 parts. A zoom/focus percentage, position in the lens file and distance from the sample.
The zoom and focus percentages are as simple as it sounds - the values that are passed from the tracking solution.
As a rule, 100% zoom is fully zoomed in and 100% focus is infinity.
This is a fraction number denoting which sample the user is editing. The denominator of this fraction is the full number of sample points across the zoom or focus. In most lens files, there are 6 zoom points and 5 focus points. However, this is not a hard and fast rule.
In Figure 30, we see 3/6 for the zoom position and 2/5 in the Focus position. This means we are editing the third zoom position, which would commonly be 40%, and the second focus position, typically 25%.
The final number for the zoom and focus data is a proximity indicator. When this value is on 1 or 0, this means we are at a sample point. In Figure 30, we have a focus proximity of 1 and a zoom proximity of 0.5 which means we are spot on on our sample point for focus, but as far as we could possibly be from the zoom sample.
View Options Panel allows the user to control viewport zoom and show crosshair (Figure 31). If viewport zoom is enabled, you use the slider to choose which part of the image to look at. The user can also click on Left, Center and Right to jump to the sides or the center of the image. Use Zoom Multiplier to specify how much zoom you need. Lastly, check Show Crosshair to show crosshair in the viewport.
Figure 31. View Options
Figure 32. Focal distance edit panel
The Focal Distance Edit Panel allows the user to edit or recreate the focal distance table for the selected lens file.
Figure 33. Focal distance units section
The user can switch between metres and feet for distances. By default, the distance data will be read out as metres.
In this part, the user can add, edit, or delete focal distance points. The user can enter a focus percentage and a distance in metres or feet, depending upon the units selected. Then, select Add to add the point. If the user selects a point from the list, this Add button will become an Edit button, allowing the user to edit a distance or focus percentage on the selected sample.
The user can also select a sample and delete a point by clicking the Delete button.
When the user has finished editing or creating their focal distance table, they can press Save. This will save the focal distance table into the lens file.
If needed, the user can also reverse the focal distance chart by clicking the Reverse Focal Distance button.
To clear all focal distance edits and restore original table, click the Restore Table button.
To delete all edits and have an empty table click the Clear Table button.
By clicking Start Focal Distance Wizard button, the user can start the focal distance wizard. This walks you through the process of creating a focal distance table.
Figure 34. Focal distance wizard window
As the user adds a sample, it gets added to the list on the right. Automatically, there will be a sample of 100% focus at 9999m. This acts as a distance of infinity.
To add a point, enter the distance in the user's specified unit and press Add. The wizard will use the focus percentage that the lens is currently at.
The user can also delete the last point they entered by pressing Delete Last.
When happy with the focal distance table, press the Done button. This will load the values from the wizard and place them into the list found in the Focal Distance Tab. The user will need to click Save for the changes to be applied.
This section will walk you through the lens tweaking process for the most common uses cases. Note: here we assume that the StarTracker is already set up and auto-aligned.
Here we explain how to setup the studio for calibration. We use 2 vertical objects (e.g. lighting stand) at different distances:
Go to Scene Setup tab (for help on how to use the interface refer to Chapter 3). Point the camera so that the crosshair is on the marker (tape) of the far object. Set Graphic 1 to Pole, Mode to Distance and enter the distance from the camera sensor to the object. Then, click the crosshair icon to place the graphic into the scene. Repeat the process with the near object in the Graphic 2 row.
Note: It is important that Graphic 1 is always the far object.
If you are tweaking a prime lens, it is best advised to use a Global Zoom tweak method. Click on Zoom 0 to open the tweaking panel. Set the focus of the lens to a position where both calibration objects are clearly visible and follow these steps:
If you are tweaking a zoom lens, Near-Wide tweak is a quick approach to making changes to the entire lens file.
However, it should only be used when, for example, a different active sensor mode is used on the camera or a minor change needs to be made to the lens file.
Steps to perform a Near-Wide tweak:
If tweaking the widest and the narrowest ends of the lens did not achieve required accuracy, more tweaks can be added. Change zoom to any position that needs tweaking and click Add Tweak. Then, repeat steps 2-9. Remember to add tweaks at the widest and narrowest ends of the lens first even if you don't need to tweak them. This will lock in the values for later when you add tweaks anywhere else in the zoom range. For best results try to add the subsequent tweaks as far away from other tweaks as possible (for example, if you have tweaks at 0% and 100% zoom, the next tweak should be around 50%, but anywhere in 40-60% range is good). Otherwise, unexpected changes can be applied to other parts of lens file. Therefore, it is a good idea to have a look at the graph to know what tweaks are to be applied to the lens file. After you have added tweaks, anywhere between the widest and the narrowest ends (tweaks number 3 or greater), you shouldn't modify the lens end tweaks again (tweaks number 1 and 2). If you want to do that, then you will have to revisit the rest of the tweaks and edit their values again or clear all of the tweaks and start again. Alternatively, Global Zoom Tweak can be used to perform tweaks at the zoom sample points. For help on how to perform it, refer to steps 1-6 in Section 5.3. Repeat the process for every zoom value in the lens file. After finishing the tweaks, don't forget to click Save. To register the tweak, click Store File in Library. To deploy to StarTracker, click Deploy to [IP].
The interface for adding the focal distance is explained in Section 4.3. There is also a walkthrough of the process in Focal Distance Wizard. It can be opened from the Edit Focal Distance section of the Lens Tweaker.
Steps for adding focal distance:
Note: Alternatively, you can use the values from the focus ring on the lens. However, use this method with caution - accuracy varies from lens to lens therefore it is recommended to measure the focal distances manually.
If Near-Wide Tweak and Global Zoom Tweak didn't achieve the desired accuracy, Advanced Tweak can be used to tweak the individual samples of the lens file. After clicking Advanced Tweak, Sample Position and Sample Proximity information will appear in the lens data panel (Figure 26). For more information about these values, see Sections 4.2.2 and 4.2.3.
The following are the steps for doing the first sample:
Now change focus until you reach the next sample. Process for tweaking the subsequent samples: