JPEG Decoder Behavior

Abbreviations

Abbreviation	Full Form
JPEG	Joint Photographic Experts Group
RGB	Red, Green, Blue
DDR	Double Data Rate
RAM	Random Access Memory
SRAM	Static RAM
VRAM	Video RAM
SOF	Start of Frame

Behavior Overview

Feature Description

The JPEG Decoder behavior allows the user to decode JPEG images at runtime using the hardware decoder. These JPEG images can be directly stored in the asset without decoding, reducing the overall asset size. JPEG images are compressed, making them smaller in size, and they can be decoded by supported platforms.

Supported Platforms

At the time of phase 1 implementation, the JPEG decoder is exclusively supported on the Infineon Traveo 2 platform.

Limitations

The JPEG Decoder Behavior currently has the following limitations under the phase 1 implementation:

Single Image Decoding: Only one image can be decoded at a time.
Preloaded Images in the asset only: The image must be part of the asset and cannot be dynamically supplied at runtime.
Baseline JPEG with RGB Encoding: The JPEG decoder behavior only supports baseline JPEG images encoded in RGB format.
Scene Editor cannot display the decoded image: The Scene Editor currently cannot display the decoded image; it is only viewable on the hardware.
CMake option “CANDERA_2D_ENABLED” must be set true for the behavior to function properly since it only supports 2D surfaces.

Using the JPEG Decoder Behavior

Import the jpeg image as raw resource: Using the import menu select “Import Raw Resources”. This ensures that the image is available in its original format for further processing
Add the Bitmap node to the scene: Add a bitmap node to a 2D scene.
Add the Behavior to a Bitmap Node: Place the JPEG Decoder behavior onto a Bitmap Node in the scene.
Set the Resource Candera Name: Paste the Candera Name of the raw JPEG image into the Resource Candera Name property using the steps below.
a. Step 1: Copy the Candera Name of the raw JPEG image from the Solution Explorer under the Asset properties of the JPEG resource.

b. Step 2: Paste the Candera Name into the ResourceCanderaName property of the JPEG Decoder Behavior in the Properties panel.

Toolchain file for the JPEG Decoder

If the JPEG decoder feature is being used, ensure that the correct toolchain file has been used. The toolchain files for the Traveo 2 platform are available in the directory cgi_studio_devices\src\Traveo2\ToolchainFiles

Use one of the following two toolchain files with the suffix _JPEG if using the JPEG decoder behavior so that the linker file dedicates a section for the JPEG Decoder output.

MULTI-toolchain-Traveo2_C_6M_JPEG.txt
MULTI-toolchain-Traveo2_C_6MDDR_JPEG.txt

The above toolchain files not only enable the creation of necessary memory sections for the JPEG Decoder but also define linker flags and macro definitions required to enable the JPEG decoder functionality in the behavior code.

Modifying the JPEG Decoder buffer memory section

The following two lines present in linker directives file create a JPEG decode buffer section to which the JPEG decoder will write the decoded image.

.jpeg_decode_buffer_base ALIGN(1) : > .
.jpeg_decode_buffer ALIGN(1) PAD(1M) : > . // JPEG decoder output

The size of the section and the location can be changed according to the target for which the application is being built.
For example, in the 6m DDR board, the location of the jpeg_decode_buffer section can be inside the LPDDR4_RAM area, however for the 6m board, it would have to be in either SRAM or VRAM (preferred).

Example for 6m board

Example for 6m DDR board

Converting images to supported format

To ensure compatibility, you may need to convert your JPEG images to an RGB-encoded baseline JPEG format. Follow the steps below to install the necessary tools and use the conversion script.

Software Requirements

The conversion script uses djpeg and cjpeg utilities, which are part of the libjpeg-turbo software. You need to install them and make them accessible via your system’s PATH environment variable.

Installation Steps

Download libjpeg-turbo from this link (libjpeg-turbo).
Extract the downloaded files, install the exe and locate libjpeg-turbo-gcc64\bin.
Since libjpeg-turbo includes the necessary executables, its bin folder path must be added to the system's Environment Variables.
Add the folder path to your system’s Environment Variables:
a. Open System Properties > Environment Variables.
b. Under System Variables, find and edit the Path variable.
c. Add the path to libjpeg-turbo-gcc64\bin.

Using the Conversion Script

The provided batch script, encode_jpeg_as_rgb.bat, automates the conversion of YCbCr JPEGs into RGB-encoded JPEGs. You simply drag and drop a JPEG file onto the script, and it processes the image automatically.

Steps for Conversion

Ensure libjpeg-turbo is installed and is accessible in PATH.
Place the batch script in a directory where your images are stored. (Code for the script is given at the end of this document in Appendix A)
Drag and drop a JPEG file onto the script.
The script will generate a new file with _rgb appended to the filename.

Example Usage

Input: sample.jpg (YCbCr format)
Output: sample_rgb.jpg (RGB format)
After running the script, the folder will look like this, with the converted images having '_rgb' appended to their filenames.

Verifying the Converted Images

To check if the image needs conversion, the following method can be used:

Install ffmpeg from the link: Download FFmpeg
Ensure the ffprobe command can be run on terminal by adding the path to the ffmpeg bin folder into the system PATH environment variable.
Open a terminal in the folder where the image is located and run the command “ffprobe image_name.jpg”.
Ensure that the format of the image as displayed by the ffprobe command is (Baseline), gbrp.

Even when run on single images, the ffprobe command will show the type as mjpeg.

Implementation Details

Methodology Overview

The behavior uses AssetProvider to fetch the required image from the asset and then passes it on to the JPEG decoder for which it waits in a busy loop.
A busy loop waiting mechanism has been used for the current implementation.
A lot of considerations need to be made for implementing a non-blocking architecture and it is planned to change the implementation of this feature from a behavior to a bitmap converter style mechanism in future releases.
Moreover, for single jpeg images of small sizes the waiting time is under 10ms.
The behavior is currently not using Infineon's JPEG Decoder Driver for the Traveo 2 since it was found during development to have many issues and does not appear to be production ready.
Therefore, register level operations have been used to directly control the JPEG decoder hardware.
Documentation of the JPEG decoder registers is available in the document TRAVEO™ T2G Automotive MCU: TVII-C-2D-6M cluster 2D registers reference manual

Implementation Structure

The general architecture of the behavior is that there is a derived class JPEGDecoder that inherits from JPEGDecoderBase class, the implementation for which depends on the target for which it is being built.
For traveo 2 6m, and 6mddr platforms, the implementation of jpeg decoder is used but for other platforms, a stub class is used.
The JPEG Decoder behavior class uses a member object of type JPEGDecoder to interface with the JPEG decoder functionality.
Once the property ResourceCanderaName is set, inside the OnChanged method, the JPEG buffer is retrieved from the asset and the JPEG header is parsed.
If the header is found to be valid and indicates that the JPEG image is in supported format, it is then passed to the JPEGDecoder object for decoding. If it is deemed unsupported for decoding, an appropriate error message is logged, and no further operations are performed.
The parser checks for the presence of SOF0 header which indicates a baseline JPEG image.
- For formats other than baseline JPEG, the SOF0 header will not be present. An SOF0 header is marked by the byte sequence 0xFFC0.
- If the SOF0 header is found, it then checks the encoded components which must be RGB, else the parser deems the image invalid for decoding.
- Finally, the parser ensures that the image width and height as encoded in the SOF0 header is within the supported range of the JPEG Decoder on the Traveo II board i.e. 1x1 to 16384x16384.
The decoder is directed to use a section in memory specially reserved for it, this is indicated by a section declaration in the linker_directives.ld file. For the JPEG decode buffer memory section to be allocated, appropriate linker flags and macro definitions must be set. This is done by choosing the toolchain files for 6m and 6m DDR boards with JPEG suffix which sets these flags.

Appendix A: Conversion Script

Paste the following code into a .bat file to create the conversion script.

@echo off
setlocal

:: Check if a file is dragged and dropped
if "%~1"=="" (
echo Usage: Drag and drop a JPEG file onto this script to convert it.
echo.
echo The output file will be saved with "_rgb" appended to the input file name.
goto end
)

:: Set the input file path
set INPUT_FILE=%~1

:: Check if the input file exists
if not exist "%INPUT_FILE%" (
echo Error: Input file "%INPUT_FILE%" not found.
goto end
)

:: Generate the output file name with "_rgb" suffix
set OUTPUT_FILE=%~dpn1_rgb.jpg

:: Temporary PPM file
set TEMP_PPM="%TEMP%\temp.ppm"

:: Step 1: Convert input JPEG to PPM using djpeg
echo Converting input JPEG to PPM...
djpeg -outfile %TEMP_PPM% "%INPUT_FILE%"
if errorlevel 1 (
echo Error: Failed to convert input JPEG to PPM.
goto end
)

:: Step 2: Convert PPM to RGB-encoded JPEG using cjpeg
echo Converting PPM to RGB-encoded JPEG...
cjpeg -rgb -outfile "%OUTPUT_FILE%" %TEMP_PPM%
if errorlevel 1 (
echo Error: Failed to convert PPM to RGB-encoded JPEG.
goto end
)

:: Clean up temporary PPM file
del %TEMP_PPM%

echo Conversion successful!
echo Output saved to "%OUTPUT_FILE%"
goto end
:end
endlocal