Harlequin RIP SDK
Outputting rasters

Raster output from the RIP is sent to the application through the raster backend API. Most of the configuration for how rasters are presented to the API is determined during the job configuration stage of processing a job, using parameters to the PostScript setpagedevice call. The raster backend selected receives API calls during interpretation of the job that can be used to prepare for the data that will be received and make some minor modifications to how it will be presented, and calls during rendering to deliver the raster data.

The application layer is responsible for selecting the raster backend API implementation to use. Different raster backends are usually used in cases where multiple output formats are supported, but may also be used to select backends for performance testing and debugging, or other reasons.

Important concepts for raster backends

There are a number of important concepts and structures to understand when working with raster data and raster backends in the Harlequin RIP.

Display lists, bands, and rendering passes

The Harlequin RIP constructs one or more display lists for each page that it interprets. A display list is an internal data structure that represents the visible content of the page, with the page objects converted into a form suitable for the renderer to operate on. The Harlequin RIP will render a display list when the page is complete. The display list structure that the Harlequin RIP uses is divided into regular bands, horizontal slices across the width of the raster output. The size of a band is very important:

  • A band is the smallest unit of output presented to the raster backend. If you have requirements for minimum or maximum memory size, data alignment, or post-processing of raster data that affects band dimensions, then you may need to control the band size precisely.
  • The amount of memory required to store the raster data for a band while rendering it is critical for performance.
  • The height of a band is important for performance. The Harlequin RIP has optimizations that use vertical coherence of objects to improve rendering performance. Selecting too small a band height means these optimisations have less effect, increasing the rendering overhead.
  • Too large a band height may reduce the number of threads the RIP can use to render a page.
  • If you are running multiple RIPs, Scalable RIP, or other memory intensive processes in parallel with the RIP, then you may need to reduce band size to avoid memory cache pressure.
  • If you have extremely wide pages that result in very small band heights, you may want to consider tiling pages. The RIP can tile some input formats, cutting individual pages into multiple smaller rasters which you will have to reassemble later. If you are using the Scalable RIP, multiple tiles from a page can be rendered in parallel.

Global Graphics have found that it is generally good for performance to keep the total memory for bands being rendered within the processor's L2 cache size, allowing some overhead for other working data structures, but also that the best way to determine band size for your RIP integration is to test with a range of band sizes on typical workflows.

The Harlequin RIP Extensions Manual documents methods to control the size of bands in the display list. The raster backend implementation can also influence the maximum size of bands, band memory alignment, and how bands are allocated.

The Harlequin RIP always renders an entire display list at a time. The Harlequin RIP may perform multiple render passes over a display list, or multiple render passes over a single band in a display list. There are a few ways in which this may become visible to you:

  • You will see calls to the raster backend API's raster requirements function for all types of render passes. These can be distinguished using the RASTER_REQUIREMENTS::render_type field. Similarly, if using a framebuffer or allocating bands yourself, you will see calls to API's the raster destination function, but these will not be followed by calls to output the band for compositing passes and partial paint passes where the SDK manages the band storage.
  • For pages that have transparency, the RIP may perform two passes over the display list to render transparent regions and non-transparency regions separately. The transparency pass is diverted into internal data structure, but if you are observing timelines for rendering, then you may observe the extra pass.
  • When using the modular halftone API for screening, the RIP may make multiple passes over a band if you have more than one halftone module selected. Each halftone module will be presented with a raster to screen, but also a mask indicating what areas of the band have data to be screened with that module. The halftone module should not modify raster data outside of the masked area it was presented with.
  • For pages that do not have transparency, the RIP may perform a partial paint to render a display list if the RIP runs out of memory. The expectation when doing this is that the rendered raster data will be written to disk or stored in some manner that allows the RIP to recover the display list memory, construct a new display list for the remainder of the page, and then read back the previously rendered raster data and render the remaining objects over the top of it. Partial paints do not work when transparency blending is required, the RIP uses different strategies to try to recover memory for such pages. The raster backend can influence whether partial paints are supported directly by the backend implementation; whether the SDK should take care of storing and reading back the data; or whether partial paints are allowed at all.

Raster layouts

The Harlequin RIP supports a considerable number of different raster output formats, each with different numbers, interleavings, and depths of data channels. The raster backend API uses a very general hierarchical data RASTER_LAYOUT structure to describe the layout of the raster data. Most of the raster backend API calls are presented with a pointer to a raster layout structure, and the core library provides calls to acquire and release references to raster layout structures, to copy, serialize and deserialize them, and to modify some aspects of them and compare some sub-structures. The RASTER_LAYOUT structure is intended to provide sufficient information and to be sufficiently general that your code can refer to it directly when needed.

Two common formats that the RIP may be configured to render output in are separations with one color per band; or as a composite with multiple colors in the each collected set of bands; but there are other layout combinations possible. In any case, the RIP outputs the bands represented by the RASTER_SHEET sub-structure as a single collection. There may be multiple raster sheet sub-structures in the entire raster layout, representing different separations of a page. The raster backend API's RASTER_START_EX and RASTER_FINISH_EX calls encapsulate output of each separation.

Another important level to understand in the RASTER_LAYOUT hierarchy is the RASTER_PLANE structure. At this level in the hierarchy the channel(s) represented by a plane are stored in a contiguous block of memory. The data in a raster plane represents a block of consecutive pixels across a number of lines. Each pixel may have one or more channels of data for it, depending on how the RIP was configured. The RASTER_PLANE and other RASTER_LAYOUT structures provide separate pixel and line strides, offsets, and storage width information to unpack or manipulate the raster data without needing to repeat the calculations the RIP performed to construct the raster.

The raster layout structure describes the structure of the raster for a page generically; the same structure will be generated for pages with the same size, output configuration, and band structure. It does not contain information that is specific to any particular instance of a page or job.

The Harlequin RIP treats the lines of raster data in bands it is outputting for each sheet as a continuous sequence over all frames in each sheet. You may need to convert these line numbers to frame indices within a sheet, and band indices within a frame. The raster layout contains fields that provide easy access to both the number of lines and bands at each level of the hierarchy, but also the total size of the raster data required ignoring separation omission, and the total size of the raster data rendered with separation omission.

Raster descriptions

The RasterDescription structure is passed to many of the raster backend API calls. Unlike RASTER_LAYOUT, this structure is very specific to the job, page, sheet, and band that is being rendered. This structure is the first place to go to find data that may be useful or relevant in identifying what output is being rendered. In particular, the job name, job number, page numbers (in various different forms), and current rendered band offsets are presented in this structure.

Raster descriptions may be packed using RasterDescriptionPack(), which can be used to serialize a raster description into a contiguous block of memory. Non-pointer fields in packed raster descriptions may be accessed, but pointer fields must not be dereferenced. Packed raster descriptions may be unpacked using RasterDescriptionUnpack(), deserializing them into a new memory allocation. Pointer fields in unpacked raster descriptions may be dereferenced safely. Unpacked raster descriptions must be destroyed when they are no longer needed using RasterDescriptionFreeUnpacked().

The current RASTER_SHEET in a raster description that is being used for rendering may be retrieved using RasterSheetFromDescription(). This is a simple convenience function that uses the raster description's sheet index to look up the sheet layout in the raster layout.

eHVD elements and backgrounds

Harlequin VariData can significantly speed up processing of variable data PDF jobs. If you intend to use External Harlequin VariData (eHVD) you need to understand eHVD elements and backgrounds, and the relationship between elements and the rasters produced for them.

Raster backends and Harlequin Parallel Pages

Some of the calls to the raster backend API are made while interpreting pages (for example, when the setpagedevice call is invoked to change page size, or when a new spot color is detected and a dynamic color separation is added), and some calls are made either just before or while rendering the page. When using Harlequin Parallel Pages, the RIP may be interpreting the next page in the job while it is rendering a previous page. Implementors need to be aware that calls to some of the API entry points may be made from both the interpreter and renderer but referring to different pages, and avoid performing operations during the interpretation calls that might disrupt simultaneous rendering operations.

Raster bounding boxes, margins and size

The relationship between the output raster bounding boxes, margins, and output raster size are shown below:

Relationship between bounding boxes, margins and output raster size

The image width and height, margins, and page relative bounding box can all be accessed from the RasterDescription structure. RasterDescription::imageWidth and RasterDescription::imageHeight are the actual size of the output raster, in pixels. The output raster size may have been restricted from the page raster set by the job or configuration if the /ImagingBBox and/or /TilingBBox keys were applied in the pagedevice (for example, when tiling output). The RasterDescription::pageRelativeBBox array gives the offsets from the output raster to the page raster, measured in pixels. The RasterDescription::leftMargin and RasterDescription::topMargin are offsets to the output raster from the output device's reference point: you would normally configure /LeftMargin and /TopMargin in the page device to shift the page raster past the unprintable area of a device. As presented in RasterDescription, the left and top margins include any extra translation resulting from restricting the page raster. The RasterDescription::rightMargin and RasterDescription::bottomMargin are offsets from the output raster to the other edge of the unprintable area of a device. The come from the /RightMargin and /BottomMargin configured in the page device, adjusted by any additional size from restricting the page raster.

Selecting raster backends

If your application will only ever use one raster output backend, then you can select it during startup using the SDK's SwLeSetRasterAPI() call.

If your application supports multiple raster output backends, the method recommended to is to select the raster backend based on the page device's /PageBufferType parameter. A more detailed implementation of the callback method presented in the startup section is explained here.

Raster backend API instances are registered using the RDR API as instances in the RDR_NAMES_RASTERAPI namespace. The SwLePgbSetCallback() function is used during startup to register a callback that will select a raster backend when /PageBufferType is changed in the RIP's pagedevice configuration:

// ... in code called after starting the SDK:
SwLePgbSetCallback("PageBufferType", ParamString, PageBufferSelect) ;
@ ParamString
DEVICEPARAM is string.
Definition: swdevice.h:1320
HqBool SwLePgbSetCallback(const char *paramname, int32 paramtype, SwLeParamCallback *pfnPGBCallback)
Sets a callback functions for the named pagebuffer device parameter.
Definition: pgbdev.c:623

The callback function used to select the raster backend interprets the pagebuffer parameter as a string, looks it up in RDR, and calls SwLeSetRasterAPI() to select it:

static HqBool RIPCALL PageBufferSelect(void *pJobContext, const void *param)
{
const char *pgbtype = param ;
// ...clrip does PGB type overrides here...
void *vptr ;
size_t apilen ;
if ( SwFindNamedRDR(RDR_NAMES_RASTERAPI, pgbtype, strlen(pgbtype),
20230105, &vptr, &apilen) != SW_RDR_SUCCESS ) {
// ...error message for no raster backend...
return FALSE;
}
sw_raster_api_20230105 *api = vptr ;
if ( apilen < sizeof(*api) ) {
// ... error message for invalid API registration size..
return FALSE;
}
return TRUE ;
}
sw_rdr_result SwFindNamedRDR(sw_rdr_namespace rdrspace, const char *name, size_t namelen, sw_rdr_id rdrid, void **pptr, size_t *plength)
Find a named RDR given the Namespace, Name, and ID.
@ SW_RDR_SUCCESS
Definition: rdrapi.h:617
@ RDR_NAMES_RASTERAPI
Definition: rdrapi.h:564
void SwLeSetRasterAPI(sw_raster_api_20230105 *api)
Sets the API structure that the RIP will call back to present raster data to the skin.
Definition: skinkit.c:920
int HqBool
Harlequin standard boolean type.
Definition: hqtypes.h:502
#define TRUE
HqBool boolean true value.
Definition: hqtypes.h:508
#define FALSE
HqBool boolean false value.
Definition: hqtypes.h:512
#define RIPCALL
The normal calling convention for RIP-exported APIs.
Definition: ripcall.h:27
A structure collecting the raster backend API functions.
Definition: rasthand.h:492

This is the method that the "clrip" application uses to select raster backends. Backends built into the "clrip" application are documented in Raster output backends.

Note
Global Graphics plans to make this method of selecting raster backends the default in a future release. When this happens, the SwLeSetPgbCallback() function will be removed, and the call used to select the raster backend based on the /PageBufferType parameter will change (it may not be exposed in the SDK).

Raster backend API

The raster backend API is represented by a structure of function pointers, called by the SDK library in response to RIP actions. A backend implemented to this API will receive calls from the RIP:

  • When the pagedevice is configured, the band size callback sw_raster_api_20230105::band_size_fn will be informed of the new can affect the band size computations.
  • The raster requirements function sw_raster_api_20230105::raster_requirements_fn is called at several points during interpretation and rendering. This function should be careful to make changes appropriate to the interpretation or rendering stage at which it is called. During interpretation, the raster requirements function is called whenever the raster layout changes: when the page device is initialized; when a dynamic separation is added because a spot color was detected in a job; and a final guaranteed call just before handing the page over for rendering. Some of the fields in the RASTER_REQUIREMENTS structure passed to this function may only be changed during interpretation, and must not be changed when rendering. The page's raster layout may change in any of the interpretation calls until the final call before rendering, but will be fixed for the final call before rendering and the calls during rendering. During rendering, there will be one call before the rendering pass, and then one call just before rendering each sheet (separation) of the page. You can determine whether a raster requirements call is for interpretation or rendering by examining the RASTER_REQUIREMENTS::render_type field: this will be REQUIREMENTS_NO_RENDER if called during interpretation, or one of the other REQUIREMENTS_render_type values during rendering, indicating the type of render pass. The first call during rendering (before the render pass) will have a NULL RasterDescription parameter; calls before rendering each sheet will have a valid RasterDescription parameter.
  • During interpretation, the blank page function is called when a page is omitted entirely, and may determine how blank pages are handled.
  • If a raster backend has indicated that it is responsible for allocating band memory, then its raster destination function will be called for each band during rendering. Note that these calls may happen before the raster start function is called.
  • The raster output start function sw_raster_api_20230105::raster_start_fn is called at the start of outputting each sheet (separation) of the page.
  • The sw_raster_api_20230105::raster_write_data_fn function is called to output each band of data when it is complete. If you are not using a framebuffer, then this function is responsible for copying or processing the raster data from the buffer provided and storing it or forwarding it to the next stage in processing. If you are using a framebuffer, this function may be empty, or may perform basic accounting such as tracking the number of lines or size of data output.
  • The sw_raster_api_20230105::raster_finish_fn function is called at the end of outputting each sheet. This is where the raster backend may pass the RIP's output to the next stage in processing, or signal that the sheet (and maybe page) is complete. You can test if the last sheet of a page has been output by comparing if RasterDescription::separationCount is equal to RasterDescription::nSeparations.
  • The sw_raster_api_20230105::job_end_fn function is called at the end of the job. This function is only called if the raster start function succeeded at some point during the job. You should use this function to clean up any per-job data or connections that the raster backend may be using.

Structured data parameters for raster handlers

Most of the raster backend API calls provide access to a structured data parameters pointer. Structured data parameters provide a method for the core library to pass relevant parameters directly to your raster backend, allowing you to modify the behavior of your raster backend based on user configuration.

Setting structured data raster parameters

Structured data raster parameters may be set during a setpagedevice call, or by using the setrasterparams pseudo-operator. In a setpagedevice call, the raster parameters are set as a sub-dictionary of the /RasterParams dictionary:

<<
/PageBufferType /TIFF
/RasterParams <<
/TIFF <<
/CompressionType /PackBits
>>
>>
>> setpagedevice

The name used inside the RasterParams dictionary must match the /PageBufferType value used to select the raster backend exactly. If the raster backend is overridden using some other method (e.g., the "clrip" -o option) or an alias is used, the raster parameters will not match. You may set raster parameters for raster backends that are not active, and select the raster backend later. You may also set raster parameters for multiple raster backends at the same time, each using the name it would be selected with /PageBufferType. When setting raster parameters during setpagedevice, the raster parameters will replace any previous parameters for the raster backend.

The PostScript setrasterparams pseudo-operator will set a dictionary of raster parameters for a raster backend, or merge them into an existing dictionary if parameters are already set. setrasterparams takes a dictionary and an optional raster backend name. The name, if present, is the PageBufferType used to select the raster backend. If the name is not present, then the current value of PageBufferType will be used:

<< /CompressionType /PackBits >> /TIFF setrasterparams
% or:
<< /CompressionType /PackBits >> setrasterparams

The dictionary of raster parameters will be merged into any existing raster parameters for the named backend, replacing any parameters with the same name. The new set of raster parameters will be provided to the next call to the raster backend (setrasterparams operates immediately, not on a pagedevice change).

Raster parameters set through setpagedevice and setrasterparams are subject to PostScript's save and restore.

Calls providing structured data raster parameters

The raster backend API band size call, the raster requirements call, and the blank page call all provide access to structured data raster parameters through their RASTER_BANDSIZE::raster_params, RASTER_REQUIREMENTS::raster_params, and RASTER_BLANK::raster_params fields. The raster output start, raster output write, and raster output finish calls provide access to structured data raster parameters through the RasterDescription::rasterParams field. The raster destination and job end calls do not provide direct access to structured data raster parameters, but access can be managed by either unpacking raster parameters in a previous call, or retaining and releasing the raster parameters appropriately.

The structured data raster parameters are represented using the structured data callback API. This API provides an abstract type sw_datum to represent data stored as dictionaries (that is, lookup tables), arrays, or the primitive types integers, real numbers, Booleans, strings, or the null value.

Retained structured data raster parameters

In all the calls where a raster backend function is presented with a sw_datum object, the object is only guaranteed to be valid for the duration of the function call. If you wish to refer to parameter values in other callback functions or threads, you must either:

  1. Unpack the parameters into variables or objects which have a suitable lifetime and scope for your use case.
  2. Or retain the structured data object, using the sw_data_api::retain() method, unpack the parameters as you need, then call a matching sw_data_api::release() method when you no longer need access to the data object.

Unpacking raster parameters

Before using the structured data API, you must get a pointer to the API implementation from RDR. If you are linking to the static SDK library, then this has already been done for you, and stored in a global variable. The SDK provides a utility function GetDataAPI() which will retrieve and cache the API pointer for you. As of Harlequin 14, this function is expected to return a non-NULL value. In previous RIPs, it may return NULL.

The raster parameters pointer provided in the raster backend API calls may be NULL if there are no raster parameters set for the corresponding /PageBufferType.

Raster backend code should test that both the API pointer and raster parameters pointer are valid before unpacking parameters, and set sensible default values in case parameters are not provided.

The structured data API contains methods to get values from dictionaries (sw_data_api::get_keyed()), to get values from arrays (sw_data_api::get_indexed()), to iterate composite objects (sw_data_api::iterate_begin(), sw_data_api::iterate_next(), and sw_data_api::iterate_end()), and to match and typecheck multiple keys in a dictionary (sw_data_api::match()).

If you are just unpacking a single raster parameter, then the sw_data_api::get_keyed() method is most likely to be useful. When using this method, you should test there was a parameter value retrieved, whether it is correct type, and if it has a suitable value:

if ( rd->rasterParams != NULL && dataapi != NULL ) {
sw_datum key = SW_DATUM_STRING("OutputBlankPages"), value ;
if ( dataapi->get_keyed(rd->rasterParams, &key, &value) == SW_DATA_OK &&
value.type == SW_DATUM_TYPE_BOOLEAN &&
value.value.boolean ) {
// ...action if param was true, i.e.,
// << /OutputBlankPages true >> setrasterparams
} // not present, not a boolean, or false
}
struct sw_data_api * GetDataAPI(void)
Get and cache the data API pointer.
Definition: skinkit.c:1058
#define SW_DATUM_STRING(_str)
Auto/static initializer for sw_datum string values.
Definition: swdataapi.h:355
@ SW_DATUM_TYPE_BOOLEAN
Definition: swdataapi.h:200
@ SW_DATA_OK
Definition: swdataapi.h:126
#define NULL
Definition of NULL pointer.
Definition: hqtypes.h:37
static sw_data_api * dataapi
The cached Data API pointer.
Definition: skinkit.c:1056
A structure containing callback functions for structured data access.
Definition: swdataapi.h:483
sw_data_result(* get_keyed)(const sw_datum *dict, const sw_datum *key, sw_datum *value)
Get a keyed value from a dictionary datum.
Definition: swdataapi.h:584
Structured data instance type.
Definition: swdataapi.h:283

If you need to unpack several raster parameters, then the sw_data_api::match() function can reduce the amount of testing you need to do, by encoding whether keys are required or optional, and the possible types allowed for the value:

if ( dataapi != NULL && rd->rasterParams != NULL ) {
sw_data_match match[] = {
SW_DATUM_OPTIONALKEY(STRING, "CompressionType"),
SW_DATUM_REQUIREDKEY(BOOLEAN, "ExpandTo8Bit"),
SW_DATUM_OPTIONALKEY(BOOLEAN, "PhotoshopFriendly"),
SW_DATUM_OPTIONALKEY(STRING, "Predictor"),
SW_DATUM_OPTIONALKEY(INTEGER, "CompressionLevel"),
} ;
if ( dataapi->match(rd->rasterParams, match, NUM_ARRAY_ITEMS(match)) == SW_DATA_OK ) {
if ( match[0].value.type == SW_DATUM_TYPE_NOTHING ) {
// ... CompressionType was not set in raster parameters
} else if ( HqMemEqual(STRING_AND_LENGTH("LZW"),
match[0].value.value.string,
match[0].value.value.length) ) {
// ...set variable for /CompressionType /LZW or /CompressionType (LZW)
} else {
// ...etc.
}
// ... Handle required Expand8Bit key and other keys
}
}
#define STRING_AND_LENGTH(s1_)
Definition: std.h:150
#define NUM_ARRAY_ITEMS(_array_)
Definition: std.h:144
HqBool HqMemEqual(const void *s1, size_t ln1, const void *s2, size_t ln2)
Compare two blocks of memory, returning a boolean indicating whether the blocks of memory are equal o...
#define SW_DATUM_REQUIREDKEY(type, name)
A key initializer macro to help build the sw_data_match structure.
Definition: swdataapi.h:923
#define SW_DATUM_OPTIONALKEY(type, name)
A key initializer macro to help build the sw_data_match structure.
Definition: swdataapi.h:944
@ SW_DATUM_TYPE_NOTHING
Definition: swdataapi.h:198
sw_data_result(* match)(const sw_datum *composite, sw_data_match *match, size_t match_length)
Match multiple data entries in a dictionary or array datum.
Definition: swdataapi.h:647
A structure used for extracting multiple values from an array or dictionary in one go.
Definition: swdataapi.h:883

Matched keys may have multiple different types, see the structured data API for details. Structured data raster parameters may have nested dictionaries and arrays.

Raster band allocation, framebuffers, and copying

The RIP normally allocates band buffers from its own memory pool. Your raster backend can change how band buffers are allocated by modifying the RASTER_REQUIREMENTS::have_framebuffer field in any of the raster requirements calls that happen during interpretation. You may not change how band buffers are stored in the raster requirements calls that occur during rendering. The values you may set this field to are:

BAND_ALLOC_RIP
BAND_ALLOC_RIP is the default value, indicating that the RIP is in charge of managing output band memory. The raster backend must copy, process, or store the data from the raster output buffer before returning control of the band buffer to the RIP (though it is possible to return the band buffer asynchronously, if your processing takes a significant amount of time).
BAND_ALLOC_FRAMEBUFFER
BAND_ALLOC_FRAMEBUFFER indicates that the raster backend will provide a full framebuffer storing all of the raster data for all sheets of a page across all render passes. By setting this value, the raster backend is guaranteeing that when the RIP asks for memory for a particular band using the API's raster destination function, the contents of the memory provided will be unchanged since the previous time the RIP requested that band. If you have the (probably large) amount of memory available to store an entire raster output page, this can be quite an efficient mode to operate in. When using a framebuffer, the raster band write function is usually empty, and the raster backend uses the raster finish function to pass the completed framebuffer off to the next stage in processing. The SDK provides some support functions to help manage framebuffers.
BAND_ALLOC_SKIN

With BAND_ALLOC_SKIN your raster backend can still take control of allocating raster output bands itself in cases where you cannot allocate and retain a full framebuffer. This may be useful if you have requirements for placement of raster bands in particular memory locations (e.g., locations where hardware DMA can access data, or memory shared with other processes).

If your raster backend takes responsibility for allocating bands, it must guarantee to provide a specified minimum number of bands simultaneously. The minimum number of bands you must provide can be found in the RASTER_REQUIREMENTS::minimum_bands field, the value depends on the number of threads that will be used for rendering, and on any halftone modules in use. If you fail to provide this number of bands simultaneously, the RIP may deadlock waiting for bands to become available.

Managing two-pass compositing or partial paint is trickier if you are managing band allocation yourself. Your raster backend will not receive raster band write calls during compositing or partial paint render passes, but it will be asked to allocate bands. You need to know when the RIP has finished using the data it stores in the band memory so that you can release those bands. This can be determined by registering an event handler for the SWEVT_BANDS_HANDLED event, and looking in the associated SWMSG_BANDS_HANDLED event message until you see a message with a line range that includes the last line of the band and has the SWMSG_BANDS_HANDLED::final boolean set to TRUE. The lines in the SWMSG_BANDS_HANDLED ranges are measured in a continuous sequence across all frames of the sheet being rendered.

If you need to allocate band buffers (or other large buffers) from the MPS arena (using SwAlloc() or MemAlloc()), Global Graphics recommends doing so during interpretation in the raster requirements call; however, care should be taken because this function is typically called multiple times during page initialization as various configuration items change the page dimensions.

Allocating memory in the raster output start call is not recommended, because the SDK only calls it when the first band is output to a page. At this point, even if the interpreter might have allocated large amounts of memory, and all available core memory may have been allocated for the band buffers. Allocating memory in the raster destination call is not recommended either, because there is no error return path to the RIP. The RIP requires that call to return a memory address that it will write data into. If an allocation fails during rendering, there are very few low-memory strategies available, so a failure is likely to raise an error and abort the job. In contrast, for allocations made in the raster requirements call, Harlequin may apply a number of different low-memory strategies to ensure that the job can run to completion.

Framebuffer support in the SDK

The SDK library provides support functions to help you implement raster output using framebuffers. The framebuffer support code enables the use of custom allocators, so framebuffer memory can come from process memory, shared memory, or your own source. Framebuffers can be acquired, released, reused between pages, and also removed from management so you can pass them to downsteam raster processing stages asynchronously.

See SDK support for framebuffer raster output.

The raster API context

The raster backend API calls are all provided with a generic data pointer. This generic data pointer is passed through from the job context parameter supplied to inputq_print_job() and to the low-level SwLeJobStart() function. You can use this parameter to pass through a job object reference from job submission to rasterization.

eHVD enabling raster backends

The core library contains a library to simplify eHVD support is raster output backends. The SDK library extends that support to enable compositing and outputting eHVD output through the same raster backend interface as non-variable jobs.

See Enabling eHVD in Raster output backends.

Blank page handling in raster backends

During interpretation, the raster backend interface's sw_raster_api_20230105::blank_page_fn function is called when a page is omitted entirely. Pages can be omitted because:

  1. There is no content in the page
  2. All of the separations in the page were omitted. All of the objects in the page are page furniture (e.g., crop marks, guides, registration marks, etc.) that were subject to separation omission, and there is no real content in any of the separations configured to be output.

The raster backend implementation may decide what action to take for a blank page, depending on your requirements:

  • You may remove blank pages from the output page numbering sequence. This is as if the page never existed in the job.
  • You may count pages in the output page numbering sequence, but omit rendering and output of the page. This is a good option if your application is able to cope with missing pages, either by inserting a placeholder for the missing page or generating a blank raster or page at lower cost than the RIP.
  • You may force the RIP to render and output the page through the normal sw_raster_api_20230105::raster_start_fn, sw_raster_api_20230105::raster_write_data_fn, and sw_raster_api_20230105::raster_finish_fn functions.

The raster backend blank page function may modify the value of the RASTER_BLANK::action field to indicate which action should be taken.

Blank pages and the Scalable RIP raster manager

When using the Scalable RIP raster manager, blank pages must either be rendered or counted, they cannot be removed. It is the raster backend's responsibility to detect this case and act accordingly. The Scalable RIP raster manager will only be active if the raster manager API is registered in RDR. You can test this using the SDK's get_skin_api_ptr() function. The "clrip" application wraps this test into a function that tests if the API is registered, and caches the result. This will not change during the lifetime of the RIP:

HqBool test_raster_manager(void)
{
static API_VERSIONED(sw_rmoutput_api,RMOUTPUT_API_VERSION) *rm_output_api = NULL ;
static HqBool rm_output_api_tried = FALSE ;
if ( !rm_output_api_tried ) {
RMOUTPUT_API_VERSION, &rm_output_api) ;
rm_output_api_tried = TRUE ;
}
return rm_output_api != NULL ;
}
// ...raster backend blank page code:
RASTER_result RIPCALL my_blank_page(void *pJobContext,
RASTER_handle *pHandle,
RASTER_BLANK *pBlank,
RasterDescription *pRasterDescription)
{
if ( test_raster_manager() ) {
if ( pRasterDescription->ripID > 0 )
// ...generate metadata...
RasterDescription rd = *pRasterDescription ;
rd.nTiles = 0 ; // Blank pages are untiled
if ( !ripfarm_report_raster(NULL, NULL, "MyOutputFormat",
metadata, pRasterDescription,
NULL, NULL) ) {
// ... cleanup and return ...
}
}
// ... rest of function ...
}
HqBool get_skin_api_ptr(sw_rdr_type apitype, sw_rdr_id id, void *papi)
Get an API pointer to a specified version of an RDR_CLASS_API resource.
Definition: rdrglue.c:289
@ RDR_API_RASTERMANAGER_OUTPUT
Definition: apis.h:69
HqnResult RASTER_result
Type for return values from raster functions.
Definition: rasthand.h:153
void * RASTER_handle
All the raster callback functions receive an opaque value of this type. The output backend implementa...
Definition: rasthand.h:130
@ BLANK_PAGE_COUNT
Definition: swraster.h:1138
HqBool ripfarm_report_raster(DEVICELIST *dev, uint8 *filename, const char *filetype, const char *metadata, RasterDescription *rd, sw_rmoutput_handled_fn *handled_fn, void *handled_data)
Send raster information for an output file to the Scalable RIP raster manager for collation.
Definition: rmreport.c:64
Structure modified by raster consumer during a DeviceIOCtl_BlankPage call to determine how to handle ...
Definition: swraster.h:1173
int32 action
Definition: swraster.h:1177
The main structure describing the raster produced by the RIP.
Definition: skinras.h:87
int32 ripID
Definition: skinras.h:167
int32 nTiles
Definition: skinras.h:172

See Enabling the Scalable RIP raster manager for details of ripfarm_report_raster() as well. Note that the number of tiles must be overridden when reporting blank pages to the raster manager, so that it expects the right number of pages. Blank pages are not tiled.

The Scalable RIP raster manager must be explicitly enabled in the Scalable RIP configuration JSON if required, by setting the UseRasterManager parameter to true.

Trim handling in raster backends

The Harlequin RIP can optimize the amount of data sent to the raster output backend by omitting bands of data that are empty. Trimming data can be turned on by setting the TrimPage boolean in the page device configuration:

<< /TrimPage true >> setpagedevice

Bands may be trimmed if they are empty, even if they are not zero. If you have configured a page background color using /EraseColor in the page device, then bands with no content will be trimmed, even when the channel data for the page background is non-zero. It is the raster backend's responsibility to reconstruct empty bands with the right background color if required for downstream processing.

The raster backend can determine whether to allow trimming, and how it operates by setting the RASTER_REQUIREMENTS::write_empty_bands field during the raster backend's raster requirements function. This field can be set during any raster requirements call until the first call during rendering a page (i.e., with a valid RASTER_REQUIREMENTS::render_type field but a NULL RasterDescription). The same value should be used for the subsequent rendering calls on that page. The values the field can take are:

  • WRITE_TRIMMED_OUTPUT indicates that the raster backend can accept output that has empty bands trimmed off the top and the bottom of the page, but interior bands should not be trimmed. If the entire page is not trimmed, then the raster that will be output will be one contiguous range of lines.
  • WRITE_NONEMPTY_OUTPUT indicates that the raster backend can accept output that has empty bands trimmed anywhere, including in the middle of the raster. The raster backend will not get RASTER_WRITE_DATA_EX calls for any empty bands.
  • WRITE_ALL_OUTPUT indicates that the raster backend cannot accept trimmed output, and all bands should be output, even if they are just the page background color. The raster backend will still be informed which bands would have been trimmed from the top and bottom of the page.

Trimming can be detected in the raster backend by examining the RasterDescription::trim_page boolean. If this is true, then the actual raster data provided may be less than the raster height indicated. The start and end of the bands that can be trimmed are provided even if trimming is not active. The RasterDescription::trim_start field indicates the first line of the raster that is actually output (or would be output if trimming is disabled), and the RasterDescription::trim_end field indicates the last line of the raster that is actually output (or would be output if trimming is disabled). Trimming operates on whole bands at a time, so these are the start line of the first band output and end line of the last band output respectively.

If the entire page is trimmed out, then the trim start will be greater than the trim end (the trim start will be the page height, and the trim end -1). In this case, the blank page handling function will be called to determine whether to output the page. If it indicates the page should be rendered, the raster output start function and the raster output finish functions will be called, but there will be no band output calls in between them. You may need to detect this case in your raster output start or finish function, and construct a blank page or placeholder if required for your downstream processing.

Separation omission handling in raster backends

The Harlequin RIP may be configured to render a subset of the colorants in the process color model, and to omit blank separations. Details of how to configure separation controls are in the Extensions Manual.

If the RIP is configured for separation omission, the raster backend will only get calls to output separations that contain marking content. The total number of separations produced, and the separation indices provided in the RasterDescription structure are adjusted to include just these separations. The raster layout structure contains the full set of separations (sheets) that would have been produced, along with their output ordering, and indices and totals noting which separations will be output and which separations will be omitted.

If all separations for a page are omitted, the blank page function will be called. The raster backend can choose whether to count the page in the page output numbering sequence, or whether to ignore it.

Asynchronous raster output

The Harlequin RIP normally outputs bands synchronously. The RIP makes a call to output a band, the raster backend processes the band, and then the RIP continues. The RIP can also perform asynchronous band output, where the RIP makes a call to output a band and then continues to render the next band while the raster backend processes the output band. This can improve performance if the raster backend has a slow processing step or stores or transmits the raster data using a slow I/O method, and can reduce latency between band output calls. There are a few steps to enable asynchronous raster output in a raster backend:

  1. The RIP must be configured to allocate extra bands. If there is no band available to render the next band while a raster backend is processing asynchronously, then there will be no performance improvement. The RIP can be configured to allocate extra bands by setting the boolean /DynamicBands system parameter to true.

    How many extra bands to allocate can be controlled if necessary by using the DynamicBandLimit system parameter. This can be used to limit the number of extra bands (setting it to a positive integer) or the size of extra bands (the absolute value of a negative integer, in KB). The default value of zero allows the RIP allocate as many extra bands as it wishes.

    <<
    /DynamicBands true
    /DynamicBandLimit -10000 % 10MB for dynamic bands
    >> setsystemparams

    The RIP needs at least one extra band available in order to make asynchronous output useful.

  2. Tell the RIP that the raster backend will be processing bands asynchronously. You can do this by setting the RASTER_REQUIREMENTS::bands_handled_by_caller boolean to false in the raster backend's raster requirements call. This indicates that the RIP (the caller) will not take responsibility for issuing the SWEVT_BANDS_HANDLED event that is used to signal that a band's output is complete.

    Note that the RIP or SDK may override this request if it is taking responsibility for band storage itself (for partial paint or two-pass compositing), but in these cases your raster backend will not receive raster output write data calls.

  3. Queue the asynchronous output bands in the raster output write data call. If the RIP is faster rendering bands than the output, it may call the raster backend's output function while it is still processing a previous band. It is the raster backend's responsibility to determine how to handle this, but a common and simple strategy is to create a lock-protected queue of bands to output, and using a separate worker thread to consume and output those bands in order. When you queue bands for asynchronous output, you must save the RasterDescription::band_owner field, the top line of the band within the entire sheet (from RasterDescription::band.y1) and the number of lines in the band (from RasterDescription::band.y2 - RasterDescription::band.y1) in your queue. These will be needed to generate the SWEVT_BANDS_HANDLED event when processing of the band is complete.
  4. Inform the RIP when band memory is no longer needed. When asynchronous processing of an output band is completed, you must inform the RIP about it. If the RIP is allocating band memory, this allows the RIP to re-use the memory allocated for another band. To inform the RIP that processing an output band is complete, the asynchronous output thread should issue a SWEVT_BANDS_HANDLED event, with an associated SWMSG_BANDS_HANDLED message containing the band owner saved from RasterDescription::band_owner and the range of lines for the band. The message should also have the SWMSG_BANDS_HANDLED::final boolean set to TRUE to indicate that it is truly finished with the band memory:

    .owner = qentry->band_owner,
    .y = qentry->y1,
    .h = qentry->lines,
    .final = TRUE,
    } ;
    if ( SwEvent(SWEVT_BANDS_HANDLED, &msg, sizeof(msg)) >= SW_EVENT_ERROR ) {
    // ... signal an error for raster finish call ...
    }
    sw_event_result SwEvent(sw_event_type type, void *message, size_t length)
    Generate an event, calling relevant handlers in priority order.
    @ SW_EVENT_ERROR
    Definition: eventapi.h:608
    Definition: swraster.h:1252
    intptr_t owner
    Definition: swraster.h:1253
    @ SWEVT_BANDS_HANDLED
    Definition: swevents.h:86

    The lines in the SWMSG_BANDS_HANDLED ranges are measured in a continuous sequence across all frames of the sheet being rendered.

The "clrip" application layer contains examples of asynchronous output in raster backends.

Raster band ordering

The Harlequin RIP normally outputs bands in order, from top to bottom of the page. If your raster backend can accept bands in any order, then the RIP can be configured to output bands in arbitrary order. This can improve performance for multi-threaded rendering. This is commonly done when using framebuffer output. The band output order can be changed in the backend's raster requirements function by setting the value of the RASTER_REQUIREMENTS::band_order field to BAND_ORDER_ANY.

Raster backend page numbering

The raster backend is provided with several different page number fields in the RasterDescription structure. These can be confusing if not used as recommended.

The output page number is the number of the page in the output sequence for a job, starting from 1. Blank pages are normally not included in this sequence, however the raster backend blank page function can be used to alter that behavior, and either output blank pages, or include the blank page in the output page number sequence but not actually output a page. The output page number should always be calculated by adding RasterDescription::pageNumber and RasterDescription::pageNumberOffset together. Do not try to use RasterDescription::pageNumber on its own: when using Scalable RIP, or multiple pdfexecid calls, or a number of other circumstances, it is incomplete.

The PDF page number is a 1-based page number for pages from PDF jobs. This is useful to determine the corresponding page from a PDF job even after selection by PageRange and reordering by PageOrder in the PDF parameters. The PDF page number also enables you to determine if a page is a blank inserted as part of a PageOrder block. The PDF page number is stored in the RasterDescription::pdfPageNumber field.

Developing raster backends for use in the Scalable RIP

The SDK's RasterDescription structure has a couple of fields to support the Scalable RIP:

ripID
ripID is zero by default (i.e., no Scalable RIP is being run) or greater than zero (i.e., 1 .. N) when multiple farm RIPs are being controlled by a Scalable RIP or RIP farm environment.
skinJobNumber
This is the Job ID created or supplied to the skin when submitting jobs through SwLeJobStart() or the HHR input queue's inputq_print_job() function. In a Scalable RIP, this is also the Scalable RIP Job ID.

If you are writing raster backend code, there will be times when you want to step through and debug your code. This can be simplified by setting the Scalable RIP global JSON configuration option FarmRIPDebugWait to true. This is because each farm RIP is its own executable with its own address space. Debugging the controlling RIP is of little use when writing raster backends.

This option will cause a farm RIP that has just been launched by the controlling RIP to wait for a debugger to connect to it before continuing. The forced break point is just after the first pass of argument parsing. This allows you to then put further break points on your own raster backend code. So, the workflow is:

  • Set FarmRIPDebugWait to true in ripfarm_global.json.
  • Start the Scalable RIP as per usual but use -nrips 1 (so you debug just one farm RIP). Although it is fine to debug multiple farm RIPs, this is probably the most useful first pass before looking into concurrency issues.
  • When the farm RIP is launched you will see that it is idle. Connect a debugger to the launched farm RIP. On Windows, the process comment should have a label such as fr001 (for "Farm RIP 1") to make it easier to find.
  • Save your project and add further breakpoints to your code.
  • In the debugger, continue with the program run. The Scalable RIP should proceed to process jobs until your own break points have been hit. You can now debug your raster backend code.

Enabling the Scalable RIP raster manager

The Scalable RIP raster manager is a component in the Scalable RIP that provides an API for clients to sequence raster output metadata. Raster backends from multiple Farm RIPs in the rip farm may complete pages out of order. Since PDF files are likely to be split amongst multiple farm RIPs, the delivery of rasters for a job are in the order of completion by the farm RIPs. Also, since the Scalable RIP can process multiple jobs simultaneously, it is possible that rasters for a later job be delivered before a job submitted prior to it. For example:

  • With 2 Farm RIPs, processing 1 page per chunk, imagine that page 1 of an 8-page job is very complex and pages 2-8 are very simple. It is possible that the rasters for pages 2-8 get delivered before page 1.
  • With 2 Farm RIPs, processing 1 page per chunk, imagine that page 8 of an 8-page job A is very complex, pages 1-7 are very simple and the whole of job B is very simple. It is possible that the rasters for pages 1-7 from job A get delivered, all of the rasters for job B then get delivered, followed finally by page 8 of job A.

It is also possible that a farm RIP fails to render a page in a job after all other pages have been rendered successfully; you cannot know if a job succeeded completely until all the rasters for a job have been delivered. Remember that page 1 could cause a rendering error after the rasters for pages 2 to N have been delivered for a particular job.

If the client wishes to consume pages in the job order, the Scalable RIP raster manager API can be used to queue the page metadata for the client.

The SDK library provides the support function ripfarm_report_raster() to inform the raster manager API about page output. This function tests whether the Scalable RIP raster manager is enabled. Global Graphics recommends calling this function in all backends you write, in case you wish to enable the Scalable RIP raster manager in future. It should be called:

  1. In the raster output finish function, when there are no errors in the output.
  2. In the blank page handling function, after detecting the RIP is a Farm RIP and changing the action to count pages.

Note that the raster manager API queues page metadata, not the page's raster data itself. Your application has to determine how raster data should be stored and transported in a RIP farm, and has to ensure that the raster data remains valid until the client consumes it. The ripfarm_report_raster() function may be supplied with a callback function to destroy or release the raster data produced by the raster backend, and a generic data pointer that can be used to identify the raster data. This callback function will be called after a client process calls the raster manager API to indicate that the data is not needed any more. The callback function will not be called on the same thread that called ripfarm_report_raster().

ripfarm_report_raster() has an optional zero-terminated C string parameter called metadata. The Scalable RIP does not impose any semantics on the meaning of this parameter. It is passed through from your raster backend to a client process that calls the raster manager API to read the page queue. Global Graphics suggests that the metadata represent a JSON string object, but there is nothing in the Scalable RIP that requires this. If the metadata you want to pass to ripfarm_report_raster() is expensive to generate, you may want to make its generation conditional on whether the raster manager is active. The method described in Blank pages and the Scalable RIP raster manager can be used to do this.

The Scalable RIP raster manager must be explicitly enabled in the Scalable RIP configuration JSON if required, by setting the UseRasterManager parameter to true. It will be enabled or disabled for the lifetime of the RIP.

Raster backend debugging

The "clrip" application layer provides assistance for debugging raster backends. You may wish to include this in your application while developing your raster output. The debug_page_info() will print the page number, size, and other details for each page output. The debugging is enabled using the /DebugPageInfo boolean raster parameter. The "clrip" application raster backends call this function in two places:

  1. In the raster output finish function, when there are no errors, e.g.:
    if ( result == RASTER_noErr )
    debug_page_info(handle->dev, handle->ps_filename, pRasterDescription) ;
    void debug_page_info(DEVICELIST *dev, uint8 *filename, RasterDescription *rd)
    If the boolean RasterParam /DebugPageInfo is true, print information about the page being output.
    Definition: debugpage.c:23
    @ RASTER_noErr
    Success.
    Definition: rasthand.h:134
  2. In the blank page handling function, if the action chosen was to count the page but not render it, e.g.:
    if ( pBlank->action == BLANK_PAGE_COUNT )
    debug_page_info(NULL, NULL, pRasterDescription) ;