Warping Overview

• Introduction
• Warping Functionality
• Using Warping with Candera
• Configuration of warping parameters
• Warping Areas
• Pixel Resolution and Number of Bits per Parameter
• Reference Points and the Warp Matrix
• Adjustments
• Rotation and Scale
• Image Flipping
• Warp Image Bounds
• Appendix A: Sample code

Introduction

The Head-up display (HUD) projects information onto the windscreen glass. An image projected to the windscreen is distorted because the glass is not flat. For correcting this image distortion, warping process shall be used and the original image shall be prepared so that the driver can observe it on the windscreen linearly.

The basic principle of the Warping Library is to map a texture onto a grid, which's form compensates for the deformation of the windscreen when the information is projected on the HUD. The texture that is used is the readily rendered content for the HUD.

CGI Studio Warping is intended to be used for correcting such image distortions. The following two components are involved in the process:

CGI Studio Controller Integration Tool

The presented figure shows an overview of the information flow when using the CGI Studio Warping components.

Three components can be identified, that together, implement the warping:

System View

Below figure shows the location of the warping library in the system and which other components use it. Globally, the warping library consists of two parts, the warping application module and a warping platform module. The platform module is platform dependent - a version for Windows host platform and a version for target hardware are available (although several platforms share the same implementation). Responsibility for each part is described in the system view.

Warping Functionality

The Warping Library supports the following functionality:

---

Warp Matrix Computation

Input:

Output:

Algorithm:

Interface:

CalculateWarpMatrix(const stc_warp_configuration_parameter_t* const pstcWarpConfigParams, const stc_warp_parameter_set_t* const pstcWarpRefPoints, stc_warp_matrix_t* pstcWarpMatrix)

---

Warp Matrix Adjustment

Input:

Output:

Algorithm:

Interface:

AdjustWarpMatrix(const stc_warp_configuration_parameter_t* const pstcWarpConfigParams, const stc_warp_adjust_parameter_t* const pstcWarpAdjustParams, en_warp_bool_t* pbRefPointsChanged, stc_warp_matrix_t* pstcWarpMatrix)

---

Reference Points Encoding and Retrieval

Input:

Output:

Algorithm:

The current active reference point shift vector set is retrieved from the warping library. Shift vectors are encoded according to the provided configuration.

Interface:

GetRefPoints(stc_warp_parameter_set_t* pstcWarpRefPoints)

---

Warp Image Bounds Computation

Input:

Output:

Algorithm:

The image area bounds are normalized as required by Candera, as it uses normalized coordinates in range [0..1] for rendering.

Interface:

ComputeWarpImageBoundsFromConfig(stc_warp_configuration_parameter_t* config, WRP_SFLOAT* x, WRP_SFLOAT* y, WRP_SFLOAT* width, WRP_SFLOAT* height)

---

Using Warping with Candera

Following are the display properties that required adjustment to use warping with Candera:

Candera::Display class uses a transaction based execution model for setting properties. Several changes can be queued. Changes have to be applied explicitly via a call to Display::ApplyChanges(). Warping matrix data from the warping library can be used directly in Candera.

Execution Overview

Warping is executed in the following steps:

Display content is rendered to a texture. This texture is clipped according to the warp image bounds and the resulting texture is mapped to the warp mesh and rendered in 3D using OpenGL and a very simple shader program.

Mesh and Texture Creation

Upon rendering of a displays content, Candera checks, if:

If any of these conditions holds, the currently used warping mesh is unloaded. A new warping mesh is computed from the warp matrix as well as the image bounds and uploaded. Vertices in the warp mesh directly map to the corresponding values in the warp matrix. Texture coordinates for vertices are computed from warp mesh vertices as well as warp image bounds. The texture is mapped to the warping mesh and distorted corresponding to the position of the of the warping mesh vertices.

---

Configuration of warping parameters

The configuration information passed to the warping library is used to check constraints and as basis for computations.

Overall configuration parameter structure

The following table details the overall configuration parameter structure and their limits.

Please refer to the following chapters for details on how the parameters influence the result of the warped image.

---

Warping Areas

The display area comprises the whole image including the grey border, as depicted in the figure below.

The bitmap image area is the area within the grey border. This is the area of interest and which should be warped. The reference points are evenly distributed over the bitmap image area.

The illuminated display area is the part of the display, which is backlit and projected to the windshield. This is the area where the warped image will be rendered into. If the bitmap image area is larger than the specified illuminated display area, then the warp image will be scaled to fit into the illuminated area boundaries.

The Warping Library needs the following parameters to be configured which are related to the warping areas:

The Bitmap Shift Area describes the connection vector between the origin of the bitmap coordinate system (0,0) and the center point of the bitmap image - the Bitmap Area Center Point. This center point represents the center point of the image area.

Example

Warp image is influenced by the size of bitmap area and illuminated area.

Reference points are distributed over display area
Display Area: 480x240
BitmapArea: 480x240 | BitmapShiftArea: 240x120
IlluminatedArea: 480x240 | IlluminatedShiftArea: 240x120

Content shrinks to fit inside illuminated area
Display Area: 480x240
BitmapArea: 480x240 | BitmapShiftArea: 240x120
IlluminatedArea: 409x190 | IlluminatedShiftArea: 240x120

Bitmap area changes, content shrinks even more to fit inside illuminated area
Display Area: 480x240
BitmapArea: 409x190 | BitmapShiftArea: 240x120
IlluminatedArea: 480x240 | IlluminatedShiftArea: 240x120

Position of the warp image inside the illuminated area can by influenced by the corresponding BitmapShiftArea parameters. Similarly, the position of the illuminated area is given by the IlluminatedShiftArea. For better visualizing this, no pre-distortion was applied.

Top left aligned
Display Area: 480x240
BitmapArea: 409x190 | BitmapShiftArea: 205x95
IlluminatedArea: 480x240 | IlluminatedShiftArea: 240x120

Center aligned
Display Area: 480x240
BitmapArea: 409x190 | BitmapShiftArea: 240x120
IlluminatedArea: 480x240 | IlluminatedShiftArea: 240x120

Bottom right aligned
Display Area: 480x240
BitmapArea: 409x190 | BitmapShiftArea: 276x145
IlluminatedArea: 480x240 | IlluminatedShiftArea: 240x120

Pixel Resolution and Number of Bits per Parameter

Pixel Resolution

The pixel resolution determines the warping accuracy. The final displacement of a reference point is the product of the warping parameter by parameter resolution. A warping parameter is therefore given by the desired displacement divided by resolution.

The resolution values supported by the warping library are: 0: ¹/₁₆ px, 1: ¹/₈ px, 2: ¹/₄ px, 3: ¹/₂ px, 4: 1 px, 5: 2 px, 6: 4 px, 7: 8 px.

Example

PixResolution: 0
Factor for multiplying: 0.0625f

PixResolution: 1
Factor for multiplying: 0.125f

PixResolution: 2
Factor for multiplying: 0.25f

PixResolution: 3
Factor for multiplying: 0.5f

PixResolution: 4
Factor for multiplying: 1.0f

PixResolution > 4
WarpMatrix is no longer computed since delta calculation leads to "out of trapezoid limit" error

Number of Bits per Parameter

The configuration parameter "NumOfBitsPerParam" gives the allowed range for the warping parameter data. Therefore, each parameter is represented as a signed integer value of the given bit width.

If n is the number of bits, then the allowed range is: -(2^n-1) .. + (2^n-1- 1).

Maximum possible value for "NumOfBitsPerParam" is 16. Considering this, the maximum allowed range for the warping parameter data is: −32 768 .. 32 767.

The length of the parameter data is given by the following formula: 2 * NumOfRefPointsX * NumOfRefPointsY * NumOfBitsPerParam / 8.

Example

NumOfBitsPerParam: 8

NumOfBitsPerParam: 12

---

 

Reference Points and the Warp Matrix

The warping parameters are based on reference points. These points are mapped by a 2D array containing coordinates inside the bitmap area to be warped.

For each reference point, a shift vector is stored by the application. Every shift vector determines the amount of distortion to be applied to the corresponding point in x- and y-direction. The warping parameter correspond to the shift vectors (in pixels) of warping interpolation point in warped images, in comparison to reference points in non-warped images (divided by the specified pixel resolution). To save memory space and computing time, the absolute coordinates are not saved as warping parameters, but rather only the distance from the reference points (without rotation and shifting).

The figure below shows a reference point grid with corresponding shift vectors.

CGI Studio Warping Library used row-major data ordering for area shift vector sets.

The warp matrix is a two-dimensional matrix computed by the warping library. Each point corresponds to a bitmap image reference point that has been shifted. The value of each point is the vector-addition of the reference point and the corresponding shift vector with regards to pixel resolution. All values of the warp matrix are normalized to the range [0..1] at the end of warp matrix computation or adjustment in order to be used by Candera.

Example

The number of reference points influence the output of the warp matrix. The points should be chosen, proportionally, with regards to the size of the bitmap area. The larger the value, the more accurate and smooth the result of the warp image.

NumOfRefPointsX: 4 | NumOfRefPointsY: 2

NumOfRefPointsX: 2 | NumOfRefPointsY: 4

NumOfRefPointsX: 4 | NumOfRefPointsY: 4

NumOfRefPointsX: 11 | NumOfRefPointsY: 5

Adjustments

An adjustment is specified by a mode, a direction and a delta. For each reference point and corresponding shift vector, the adjustment is computed and applied. The resulting shift vector is again cached in the warping library. This way of implementation allows incremental adjustments.

Mode defines the way of the distortion. Following modes are available:

Direction specifies the direction in which the distortion is applied. Following directions are available:

Delta specifies the amount of distortion that is applied. Delta is an unsigned integer value in range 0..255 (1 byte). The delta is added to the existing offset of the corresponding reference points.

Predefined Adjustments

Based on mode and direction, the following transformations are possible:

Formulas for adjustments

The following formulas describe the mathematically necessary adjustments of the reference points. If the requested adjustment exceeds the limit, no calculation shall be done and the previous status will be kept.

Trapezoid mode

Parallelogram mode

Pinbalance mode

Pincushion mode

Size mode

Rotation mode

Shift mode

Rotation and Scale

Rotation

The rotation takes place around the warped center point of the bitmap area.

If an uneven number of reference points in x/y direction is specified, the warped center point coincides with the middle reference grid point and the coordinates of the rotation axis are present. If an even number of reference grid points is use, the coordinates of the warped center point are calculated.

An initial RotationValue and the RotationLimit are specified through the configuration parameter structure:

Warp image rotation at run-time is achieved through the adjustments supported by the Warping Library. Subsequent calls of warp matrix adjustments result in incremental rotation of the warp image with an incremental step equal to the specified delta value.

Example

RotationValue: 500 | Rotate 5° to right	RotationValue: -1000 | Rotate 10° to left
RotationValue: 3000 | Rotate 30° to right	RotationValue: -4500 | Rotate 45° to left

Scaling

The warping library performs automatic rescaling of the resulting warping matrix whenever it overlaps or exceeds the boundaries of the illuminated area. It is possible to identify two cases in which rescaling will be done:

The ScaleMode parameter influences the behavior of scaling by modifying the internal scale factor:

Example

No Scaling occurs BitmapArea: 408x190 | Illuminated Area: 480x190	Scaling occurs because bitmap area exceeds illuminated area BitmapArea: 408x190 | Illuminated Area: 375x190
No Scaling occurs since warp image does not exceed illuminated area BitmapArea: 408x190 | Illuminated Area: 480x190 RotationValue: 0° | RotationLimit: 45° ScaleMode: 0	Scaling occurs because of specified RotationLimit and ScaleMode BitmapArea: 408x190 | Illuminated Area: 480x190 RotationValue: 0° | RotationLimit: 45° ScaleMode: 1
Scaling down during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 0	Scaling down during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 0
Scaling down during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 0	Scaling down during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 0
No scaling during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 1	No scaling during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 1
No scaling during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 1	No scaling during incremental rotation BitmapArea: 408x190 | Illuminated Area: 480x190 ScaleMode: 1

---

Image Flipping

Flipping transforms the content prior to warping the image on screen. Flipping is supported only when warping is enabled. The following orientations are supported:

Changing the image orientation is achieved through the public interface of Candera::Display class using Candera::Display::SetWarpOrientation, like in the following snippet.

    // flip the image vertically and horizontally
    display->SetWarpOrientation(Candera::DisplayOrientation::VerticallyAndHorizontallyFlipped);

Example

 

Warp Image Bounds

Warp image bounds comprise upper left corner point of the bitmap image area, its width, and height (normalized to the range [0..1]).

They are used to specify to Candera which part of the rendered display image shall be mapped to the warping mesh computed from the warp matrix. If not specified, then the complete display image will be mapped.

The following code snippet shows how to retrieve the warp image bounds from the Warping library. These can be further passed to Candera through the public interface of Candera::Display class using Candera::Display::SetWarpImageBounds.

    WRP_SFLOAT textureOriginX; 
    WRP_SFLOAT textureOriginY;
    WRP_SFLOAT textureWidth;
    WRP_SFLOAT textureHeight;
    
    // retrieve warpTextureBounds
    ComputeWarpImageBoundsFromConfig(&stcWarpConfigParams, &textureOriginX, &textureOriginY, &textureWidth, &textureHeight);
    Candera::Rectangle warpTextureBounds(textureOriginX, textureOriginY, textureWidth, textureHeight);
      
    // set the bounds of the warping texture  
    display->SetWarpImageBounds(warpTextureBounds);

Example

Appendix A: Sample code

Please refer to the following code snippets for information on how to the Warping Library and Candera are used in order to calculate the warped output that will be rendered on display.

Create Warp Matrix

    en_warp_result_t enResult = WarpResultSuccess;          // Enumeration for possible return values of warping library functions.
    stc_warp_configuration_parameter_t stcWarpConfigParams; // Warping configuration parameters and limit parameters.
    stc_warp_parameter_set_t stcWarpParamSet;               // Structure of the encoded input parameter set for the reference points.
    stc_warp_matrix_t stcWarpMatrix;                        // Structure for the calculated matrix vertex points.
    
    ... // Specify the configuration parameters and the input parameter set according the project needs
    
    // generate the warp matrix based on provided input configuration data
    enResult = CalculateWarpMatrix(&stcWarpConfigParams, &stcWarpParamSet, &stcWarpMatrix);

    // check return error of previous call
    if (enResult == WarpResultSuccess) {
        FEATSTD_LOG_DEBUG("Warp matrix succsessfully created.\n");
    } else {
        FEATSTD_LOG_DEBUG("Failed to create warp matrix.\n");
    }

Warp Matrix Adjustments

    en_warp_result_t enResult = WarpResultSuccess;          // Enumeration for possible return values of warping library functions.
    stc_warp_configuration_parameter_t stcWarpConfigParams; // Warping configuration parameters and limit parameters.
    stc_warp_adjust_parameter_t stcWarpAdjustParams;        // Structure with all parameters for delta calculation.
    stc_warp_matrix_t stcWarpMatrix;                        // Structure for the calculated matrix vertex points.
    en_warp_bool_t bSuccess;
    
    ... // Specify the configuration parameters
    
    // Specify parameters for delta calculation (example)
    stcWarpAdjustParams.WarpAdjustMode = WarpAdjustModeTrapezoid;
    stcWarpAdjustParams.WarpAdjustDirection = WarpAdjustDirectionDown;
    stcWarpAdjustParams.WarpDelta = 100;
    
    enResult = AdjustWarpMatrix(&stcWarpConfigParams, &stcWarpAdjustParams, &bSuccess, &stcWarpMatrix);

    if (enResult == WarpResultSuccess) {
        FEATSTD_LOG_DEBUG("Adjusted warp matrix.\n");
    } else {
        FEATSTD_LOG_DEBUG("Failed to adjust warp matrix.\n");
    }

Create the Candera WarpMatrix from the previously calculated warp matrix

      // Create warp matrix. This is a proof of the disposer concept.
      // The matrix data doesn't need to be copied, since it is static and can
      // be simply set into the warp matrix without a disposer.
      Candera::Float* matrixData = CANDERA_NEW_ARRAY(Candera::Float, 2 * stcWarpMatrix.usNumRefPointsX * stcWarpMatrix.usNumRefPointsY);
      if (matrixData == 0) {
        FEATSTD_LOG_ERROR("failed to allocate matrix data");
        return;
      }
      Candera::MemoryPlatform::Copy(matrixData, FeatStd::Internal::PointerToPointer<void*>(stcWarpMatrix.pVertex), 2 * stcWarpMatrix.usNumRefPointsX * stcWarpMatrix.usNumRefPointsY * sizeof(Candera::Float));
      typedef Candera::MemoryManagement::AdaptedArrayDisposer<Candera::WarpMatrix::ConstData, const Candera::Float*> MatrixDataDisposer;
      Candera::WarpMatrix warpMatrix(stcWarpMatrix.usNumRefPointsX, stcWarpMatrix.usNumRefPointsY, matrixData, &MatrixDataDisposer::Dispose);

Calculate Warp Texture Bounds

      WRP_SFLOAT textureOriginX; 
      WRP_SFLOAT textureOriginY;
      WRP_SFLOAT textureWidth;
      WRP_SFLOAT textureHeight;
      
      ComputeWarpImageBoundsFromConfig(&stcWarpConfigParams, &textureOriginX, &textureOriginY, &textureWidth, &textureHeight);
      Candera::Rectangle warpTextureBounds(textureOriginX, textureOriginY, textureWidth, textureHeight);

Render Warped Image to Display

      // define display parameters
      Candera::Display::CommonSettings params;
      Candera::MemoryPlatform::Set(&params, 0, sizeof(params));

      params.width =          800;
      params.hps =            832;
      params.hpe =            976;
      params.hpt =            1008;
      
      params.height =         600; 
      params.vps =            612; 
      params.vpe =            618; 
      params.vpt =            631;
        
      params.refreshRate =    60;
      params.colorBits =      32;
      
      // create display
      Candera::Display *display = Candera::DevicePackageInterface::CreateDisplay(0);
      if (display == 0) {
          FEATSTD_LOG_ERROR("Failed to create display.");
          return false;
      }
      
      //enable warping
      display->SetWarpingEnabled(true);
      
      // pass previously created Candera warpMatrix to display
      display->SetWarpMatrix(warpMatrix);          
      
      // optionally, set the warp texture bounds
      display->SetWarpImageBounds(warpTextureBounds);
      
      // ApplyChanges() has to be called for the changes to be taken into account
      display->ApplyChanges(); 
      
      //upload display
      if (!display->Upload(params)) {
          FEATSTD_LOG_ERROR("Failed to upload display.");
          return false;
      }

Warping should be enabled before the NativeHandle is attached to the display (attached either by AttachNativeHandle or created at display upload).

Create a Default WarpMatrix Data

      // retrieve display
      Candera::Display *display = Candera::DevicePackageInterface::GetDisplay(0);
      
      if (display == 0) {
        FEATSTD_LOG_ERROR("Failed to retrieve display.");
        return false;
      }      
      
      // default warpMatrix data
      static const Candera::Float data[] = {
                0.0F, 0.0F,
                1.0F, 0.0F,
                0.0F, 1.0F,
                1.0F, 1.0F,
            };
      
      Candera::WarpMatrix warpMatrix(2, 2, data);
      display->SetWarpMatrix(warpMatrix);
      
      Candera::Rectangle defaultImageBounds(0.F, 0.F, 1.F, 1.F);
      display->SetWarpImageBounds(defaultImageBounds);