|
Harlequin RIP SDK
|
|
|
|
Harlequin RIP SDK
|
|
The Harlequin RIP has a number of methods to monitor its progress and status when processing jobs. Some of these methods provide detail in machine-readable forms that can be parsed, logged and presented in user interfaces; other methods may only be suitable for log capture and human interpretation. The methods include:
As the Harlequin RIP runs, it outputs messages containing information about the RIP's configuration, operation, and the jobs it is processing. This information comes from a variety of sources:
%stdout% and %stderr% file objects are connected by the interpreter to the message channels, and so data written to these file objects will appear in the RIP output. The PDF interpreter extracts metadata from PDF jobs and announces the PDF version, job title, creator, creation data and other details.The "clrip" application layer can route the output messages from the RIP and SDK to one or more destinations. All messages are normally sent to the standard output stream of the application.
Messages can also be appended to a log file containing all messages for all jobs processed by using the -X command-line option. This log file is in the RIP's SW data directory, called SW/LE_LGFLE. If you are using a different writable data directory (using the -W command-line option), then the log file will be in that directory instead. This log file is never truncated by the RIP and it will accumulate over multiple runs of the RIP. If you are capturing this log file or using a similar method in a production application, you may want to delete or rotate the log file. Rotating the log file cannot be performed while the RIP is running. The -X option can be used with Scalable RIP, but note that there will be a separate log file for the controller RIP and each Farm RIP, in their respective data directories. The controller log file will show processing information for job submission and then feedback from the Scalable RIP's progress monitor, each Farm RIP log will contain a subset of the page information for the entire job.
Messages can be appended to a separate log file for each job, using clrip's -L option. This appends message output to a log file located in the same directory as the input file, which makes it unsuitable for use with some input sources such as hot folders. The log file name is constructed by appending .log to the input file name. These log files are not deleted or rotated by the RIP either, but can be deleted by the application or another process once the job is completed.
Both log files can be enabled at once, if desired.
Error output is normally sent to clrip's standard output for compatibility purposes, but can be sent to its standard error by using the -err command-line option.
Global Graphics strongly recommends that you do not try to parse log file output for data to present in user interface or use in your application. There are many problems with parsing log files, which can be solved by using the monitor events instead. Among the problems are:
Monitor information output from both the core library and the SDK is routed through monitor events. Monitor events carry metadata about each message, including a related timeline, the output channel, and the monitor information type (including an error code and in many cases a machine-parsable unique ID for the message). This is useful when you are performing complex integrations. Individual messages can be diverted, suppressed, or used to trigger warnings or alerts in your DFE or GUI.
The event system is available from starting the SDK for the entire duration until the SDK is shutdown. The SDK now allows installation of a default monitor callback function using SwLeSetMonitorFunction() that can be used to capture and contextualize monitor information that is not presented through events.
All messages sent from the core library, or from the SDK functions SkinMonitorf(), SkinEMonitorf() or SkinVEMonitorf(), are formatted into a message buffer, and then issued through a SWEVT_MONITOR event. This event has an associated SWMSG_MONITOR message. The SWMSG_MONITOR message contains a pointer to the message buffer, the length of the text in the message buffer, a timeline reference, a monitor channel, and a monitor type.
The message text is expected to be in UTF-8, potentially with a byte-order mark in front. However, since it may be generated from job output, it may potentially be badly formed, or have not been transcoded from a job-specific encoding. The message text may also not be zero-terminated, or may include internal zero bytes, so the message buffer length should always be used to determine how much valid data is in the buffer. If there is a byte-order mark at the start of the message buffer, then the client may use this to convert from UTF-8 to a desired encoding, but should check for errors in transcoding.
The timeline reference in the event message may provide a useful reference to associate the message with the process within the RIP or SDK that produced it. Not all messages have a valid timeline reference.
The channel identifier in the event message separates the routes that messages may take through the message system. There are a set of predefined channels that the RIP uses in SW_MON_CHANNELS, but you can add additional channels in your code. The channel may be used to separate the destination of messages at the last stage of the message system. For example, the "clrip" application may divert messages sent to the MON_CHANNEL_STDERR channel to the application's standard error stream, or may colorize messages for different channels when using the -! command-line option.
The message type in the event message is a field that combines the class of the message (is it an error, a warning, informational, or debug?) with a sub-categorization and in some cases a unique ID that identifies the exact semantic meaning of this particular message.
The event's channel and type (class and ID) should be used to distinguish messages and decide how to handle and respond messages in preference to parsing the message text.
If all that you require is to capture the entire message stream and route it into your application, then installing a message capture function with SwLeSetMonitorFunction() is sufficient. This will only give you access to the message text, and not the timeline reference, channel or type. If you need access to the timeline reference, channel, or type, then you will need to install an event handler for the SWEVT_MONITOR event. Event handlers can be installed individually using SwRegisterHandler(), and in groups using SwRegisterHandlers():
The example above show two event usage patterns that are useful with monitor events. The event handler declared as monitor_filter() looks at a message, decides whether to filter it in some way, and either modifies the message or sets state that will be used by a different layer to determine what to do with the message. It passes the event on to lower priority handlers, and then restores any state it modified when filtering. This is the method used by the "clrip" application when diverting output for the MON_CHANNEL_STDERR channel to the standard error if the -err command-line option is specified. The event handler declared as monitor_last() stops any lower-priority event handlers from running. This is the method used by the SDK's fallback message handler.
The priorities that should be used by monitor event handler are complex, and are explained in a separate section.
The "clrip" application uses C standard I/O at the last level of handling the monitor output, either from the events or from the monitor callback functions used to write data from the RIP or skin, and in some other locations before the skin is fully booted and for some internal debugging information. It is easy to divert output messages from the SDK and the RIP to other log files or log systems uniformly by reconfiguring a monitor callback function and/or installing event handlers.
The "clrip" application uses monitor event handlers to implement the color output console selected using the -! option. If you use that option you will notice some extra timeline information for the skin and skin job timelines, and also some messages appearing in different colors because they are routing to different channels internally.
The clrip skin normally runs in a compatibility mode, where output to the core's %stderr% stream is routed to the same output channel as the standard output, but skin calls to output to standard error are routed to the standard error channel. The -err option removes the compatibility routing and sends all %stderr% output from the core and the skin to standard error.
The monitor event system is active from starting the SDK to stopping the SDK. Any messages output using SkinMonitorf(), SkinEMonitorf() or SkinVEMonitorf() before starting the SDK or after stopping the SDK will be sent directly to the default message handler function set by SwLeSetMonitorFunction(), or will be ignored if no default handler function has been set.
All of the message formatting routines in SDK use the RIP core's swncopyf() call, so all the formats supported by it can be used.
The monitor event interface defines a hierarchy of priorities, which are suggested for use when installing handlers. The core library and SDK only register handlers at a couple of levels, but understanding what the priorities are intended for may help you if you register your own monitor event handlers.
Most handler functions should pass monitor messages to lower priority handlers, by returning SW_EVENT_CONTINUE from the handlers. Monitor event handler functions that augment the message or the environmental state for processing the message should modify state, call SwEventTail() and capture the return value, restore state, and change the return value to SW_EVENT_FORCE_UNHANDLED if the SwEventTail() call returned SW_EVENT_UNHANDLED. This is the normal event handler filtering pattern.
The skin is expected to install its normal output handler at priority MON_DEFAULT_HANDLER. The "clrip" application color console handlers installed when using the -! command-line option are installed at this priority. The "clrip" application handler to direct MON_CHANNEL_STDERR output to the standard error stream is registered at the MON_PRE_DEFAULT priority, just above the color console handlers.
The SDK may install an event handler to filter the progress messages that will be output at the MON_FILTER priority. Filtering out messages should be done at this relatively high priority, so that lower layers don't waste time processing messages that will be discarded.
The core library registers some handlers at the very low MON_COMPATIBILITY priority. These handlers are only called if a default handler is not installed by the application. They convert unhandled monitor event messages for progress and halftone channels to output to files on the legacy progress device. The core library also registers a handler at the even lower MON_FALLBACK priority. This is called only if a default handler is not installed by the application. It converts all remaining monitor event messages to output to an unnamed file on the legacy %monitor% or %console% devices, if either are mounted.
The "clrip" application installs its own fallback handler at a lower priority than the core's MON_FALLBACK handler, in case neither %monitor% nor %console% devices are mounted. This is the handler that will direct any messages to the callback function registered using SwLeSetMonitorFunction().
Error messages are issued by the RIP when an error is handled by the RIP's PostScript error handler. Errors will normally cause a job to be aborted, but the error handler can be overridden to suppress some errors. In this case, it is possible to see a monitor event message indicating an error even when the job does not terminate. Whether the job is terminated may also be affected by the error detail event handling.
The event message's type field can be decomposed into the class, sub-class, and for errors the error number and ID, as detailed in the sw_mon_type documentation. If the ID part of a message's type is non-zero, the entire type field can be treated as a unique identifier for that particular message and its root cause.
Not all errors or messages in the RIP have unique IDs (for those that are not unique, the ID part of sw_mon_type is zero). For memory allocation errors and interrupt errors, this is deliberate. The exact location that an interrupt or memory error is triggered is generally not informative, since it is related to a condition external to the code triggering the error, and may not be repeatably triggered at that location. Global Graphics is working on expanding the use of error IDs where it will be informative.
The distribution contains a CSV file with the unique IDs and their associated names and message reasons (monitoruids.csv), and also an HTML file with the unique IDs and their associated names and message reasons sorted both by number and by name (monitoruids.html, also here). Global Graphics has procedures in place to ensure that unique IDs for a particular root cause do not change between releases, and that obsolete unique IDs do not get re-used for different purposes. Comparing the unique IDs listed in these files against the monitor event message's type field is recommended as the most reliable method of identifying specific output messages.
Some of the monitor event messages are output by the core library to provide job timing information. When a job is processed, information of the following type is displayed in the command window:
The lines with timing information may be filtered out if you do not want this information to appear in the log. The SDK provides the progevts_suppress_types() function to filter out the types of message that you do not want to see.
Timelines can be used to detect and measure the start and end of many processing steps in the RIP, and also to get meaningful indications of progress on many of these processes. A timeline represents an object or a process that has a finite life. Timelines are created by some processing steps when they start, updated with progress as they proceed, and either ended or aborted when they finish.
Timelines may be monitored through the use of timeline events. Timeline events are all associated with a SWMSG_TIMELINE message. This message contains, amongst other fields, the timeline reference, its parent timeline in the hierarchy, its state, the unit of progress, the start, end, and current progress position of the timeline, and a name or title associated with the timeline. Timelines may have context data associated with them, allowing any code with a timeline reference to attach a data pointer to a timeline that it can retrieve later. This is useful for maintaining an association between a timeline and objects represented in your application (for example, a reference to a progress bar in a user interface might be attached to a timeline).
Events are issued when timelines are started, ended, or aborted; when progress is recorded on them; and when the timeline title is changed. Timelines may be bounded or unbounded in progress, depending whether they represent a process with a known limit or not. The timeline extent may be modified for some timelines, which will issue another event type.
Each process or entity represented by a timeline will have a unique timeline reference. If there are multiple entities or processes of the same type running at the same time, then they will have different data associated with each different timeline reference.
To monitor progress using timelines, you need to register event handlers for the appropriate timeline events. You will probably want to register events for timeline start, end, and abort, and maybe also for progress, extent changes and maybe for title changes:
Each timeline handler should test the timeline message to ensure it is operating on the correct timeline type, using the SWMSG_TIMELINE::type field. It is common to share the same handler for similar types of handler, such as timeline end and timeline abort events, or the timeline progress and extent events. You can distinguish the use cases by examining the event type field in sw_event::type. These fields have the same name, so be careful to ensure you are testing the correct structure:
Event handlers should be deregistered if you no longer need them:
The description of each timeline in Harlequin Core Timelines or SDK and Skin timelines indicates what process or entity it represents, and the units of progress (if any) that are reported for it. The units of progress may be bytes, pages, lines, bands, or other units, so be careful to understand what is being represented.
Timeline events may also be used to trigger destruction of objects at suitable points. For example, skin code that allocates memory from the core library's memory pools may need to perform cleanup actions before the RIP is shut down. The SWTLT_CORE_READY timeline provides a convenient hook to monitor when the RIP is about to shut down, and to remove any references to core library memory pools.
The timelines defined by the core library are described in Harlequin Core Timelines. The well-known timelines used by the skin are described by SDK and Skin timelines. You may create your own timelines too.
The timelines that are probably of most interest for monitoring progress are:
Monitor event messages for errors may be output when an error is handled even if it does not terminate a job, so they are useful for capturing error detail, but not a definitive indication that a job was terminated. A different method should be used to confirm whether a job was terminated due to an error, such as monitoring the job timeline or monitoring the error events.
The events SWEVT_INTERPRET_ERROR or SWEVT_RENDER_ERROR are issued if a job fails, and the normal PostScript error handling did not suppress the error. A SWMSG_ERROR message is associated with both of these events. These events with their message act both as a notification about an unhandled error, information about the error detail, and a last-chance control to indicate how to report or handle the error.
The SWMSG_ERROR message contains a timeline reference of the component generating the error; the page number on which the error occurred; the error UID, error name, command, and detail that were provided to the PostScript error handler, and two boolean fields indicating whether the job should be marked as failing; and whether normal error reporting should be suppressed.
The job may already have been aborted by a different error, so setting the job failure boolean to FALSE is not guaranteed to ensure it continues.
If the SWEVT_INTERPRET_ERROR event is issued, the job has already been terminated, so setting the suppress handling boolean just suppresses the normal error report. If the SWEVT_RENDER_ERROR is issued, setting the job failure boolean to FALSE and suppress handling to TRUE may allow subsequent pages of a job to continue outputting.
If no default monitor event handler is installed by the application, the core library's compatibility layers may redirect messages to files on some legacy devices, if they are mounted. These methods of reporting progress are compatibility methods for legacy integrations only, and should not be used for new integrations. Install a default monitor event handler instead.
If no monitor event handler handles a message before the core library's ::MON_COMPATIBILTY priority handlers, messages on the progress channel will be directed to the %progress%JobLog file, and messages on the halftone channel will be directed to the %progress%HalftoneInfo file. If a device is mounted using the %progress% name, then these files will be opened, written, and closed for these messages.
If no monitor event handler handles a message before the core library's ::MON_FALLBACK priority handlers, messages will be directed to an unnamed file on the %monitor% or %console% devices. If a device is mounted using the %monitor% or %console% name, then an unnamed file will be opened, written, and closed for all remaining messages.
The timing messages produced by the RIP have a format that includes both a human-friendly and a machine-parsable part (as shown in Progress events). If you want to output messages using the same format as the RIP, you can call the core library utility function swncopyduration(). This is the same function as the RIP uses to format durations specified in milliseconds to include in output messages.
The console and job log output from the Scalable RIP differs in the detail from a single RIP:
Jobs are submitted and processed asynchronously, so the Total time reported by the RIP is just the time required to submit the job to the Scalable RIP's job controller, and is reported before the job is actually rendered. The Scalable RIP reports:
On completion, it reports three times:
In the Scalable RIP, Farm RIP monitor output cannot be seen by default. This can make the Scalable RIP appear to be doing nothing for large jobs where all RIPs are busy processing a single job for a long period of time. The controlling RIP will report progress made by the Farm RIPs, to give the user feedback that the RIPs are making progress. Page progress monitoring is time based and is controlled by the global JSON configuration. For example, progress lines look like the following:
By default, a progress line will be output every 1000 milliseconds (1 second). This default value can be changed by modifying the value associated with the key ProgressReportMillisec in the global configuration JSON (usually in the SW/RIPFarmConfig/ripfarm_global.json file).
On MacOS and Linux, when Farm RIPs are launched by the controlling RIP, the standard out, standard error, and the standard input file descriptors are redirected to /dev/null. This means you will not see any output from the Farm RIPs directly. Should you wish to see monitor output from all the farm RIPs you can set the global configuration option FarmRIPOpenConsoleWindow to true. This means that the Farm RIPs inherit the standard input, output, and error file descriptors from the controlling RIP when they are forked and execed. Doing this results in you seeing all the interleaved monitor output from all RIPs, which can sometimes be useful for debugging or trying to find issues, but is difficult for users to comprehend.
On Windows, console applications work differently to UNIX-based systems and each Farm RIP launched gets its own console to interact with. By default, these consoles do not have a window. This means you will not see monitor output from the Farm RIPs on Windows by default. As a debug aid, you can use the global configuration options FarmRIPOpenConsoleWindow and FarmRIPOpenConsoleMinimized to see these console windows. The console Windows are named carefully so that you can set properties on these console Windows to save their position, size and color.