Log in

System

You are here: System » PublishPlugin

PublishPlugin

31 January 2018 - 06:24 | Version 1 |
Generates a static view of a web, as HTML files on disc, or as a PDF, or as a zip or tgz archive file, or by uploading directly to an FTP server.

This is the most complete publishing solution for Foswiki.

publish.gif PublishPlugin provides support for the generation of stand-alone HTML from a Foswiki installation. It will generate fully rendered versions of a set of Foswiki pages together with any attached files, and (optionally) any external resources referenced.

When you want to read a document stored in Foswiki, you have to have access to the web server that hosts that document. There are times when this may not be desirable, or even possible. For example:
  1. Foswiki is used to create documentation which has to be readable off-line
  2. Published versions of Foswiki pages must be read-only
  3. The Foswiki server is inaccessible to the audience (e.g. on the other side of a corporate firewall)
  4. You want an efficient, high-density snapshot of the wiki content

To address these requirements the PublishPlugin supports the generation of several different document formats from Foswiki pages, including HTML and PDF.

Features

  • All standard Foswiki macros are interpreted, and plugins are called
  • Powerful support for choosing what content gets published
  • Full support for hierarchical webs
  • Any links to the 'pub' areas of topics in the web are automatically resolved, and the referenced files copied
  • Any links to resource outside the wiki are resolved, and a snapshot of the resource is stored in the output
  • Output in HTML or PDF
  • HTML can be compressed in different archive formats
  • Incremental output of HTML for efficient incremental publishing
  • Format-specific skins (such as viewpdf) can be specified
  • Able to upload directly to a remote server via ftp
  • Complete history of what was published, and when
  • Fully compatible with BookmakerPlugin

Usage

The plugin can be used in a number of different ways:
  • Via the Publish Form on this page (recommended)
  • From the command line (recommended)
  • From a re-usable configuration topic
  • Using BookmakerPlugin (if installed)

Publish Form

The quickest way to publish is by filling in the following form.

The output is always generated in a directory designated during installation. Admins can use the PublishPluginControlCentre to manage the published output.

Publishing is an access-controlled process; before you can publish, you have to have VIEW access to the topics you want to publish (and CHANGE access to the publishing history topic, if you have asked for one).

You can also create a permanent topic in a web to help with a repeated publishing process.

HELP Most publishing tasks will only require setting the immediately visible options. Other more advanced options can be accessed by opening the collapsible sections.

Choose what to publish Parameter Name
Topics The topics parameter is used to give a (comma-separated) list of web.topic names. These can be specified using wildcard patterns.
  • Myweb.* will publish all topics in the Myweb web (but not in subwebs)
  • Myweb*.* will publish all topics in the Myweb web and all its subwebs
  • *.* will publish all topics in the wiki
  • Web. implies Web.*, and .Topic and Topic both imply *.Topic
  • The list is expanded in left-right order. You can edit the list at any point by prefixing an entry with a - sign e.g. *.*,-*.Last* will publish all topics in the wiki in web.topic order except topics starting with Last
  • You can use - to control the ordering; *.*,-*.Last*,*.Last* will do the same, but followed by all topics in the wiki starting with Last
  • If a topic is matched twice in the ordering, it will be published twice (this will be a no-operation for most types of output, but may be useful in PDF)
  • The order in which wildcards are expanded is defined by the locale collation.
These topics are only the start points for publishing. Links to other topics in the wiki will automatically be followed, and those topics published as well. 		topics
Unpublished Topics Sometimes links to topics that are not selected by the topics parameter may be found during publishing. You can choose how these links will be handled:
  • rewrite (the default) - the link will be processed as if the topic was being published
  • follow - the link will be rewritten and the most recent version of the topic referred to in the link will itself be published,
  • 404 - the link will be rewritten to ensure it is broken,
  • ignore - the link will be left as a link to the wiki. Note that in most cases this will leave a broken link, but may be useful for certain server configurations.
unpublished
Versions Topic Name of a topic in each published web that contains a table, each row of which maps topic names to the version of that topic to publish.
The table can be generated by a %SEARCH{}% or other macro. For example: |Web.TopicName|1.33|.
If a topic does not appear in the table, the most recent version will be published. 		versions
Content Filter A regular expression that will cause a topic to be excluded if the expression matches the topic content. You can use a simple string here, which will be matched exactly, or you can read up on perl regular expressions on the web. rexclude
Processing options
Default processing of topics for publishing tries to render the topics as closely to the way they are viewed in the wiki as possible. These options provide a finer level of control.
Publish skin Setting of the SKIN preference to be used when topics are published. See Skins for more informations on skins. You can pick basic_publish (a very, very simple skin), or plain, or a print skin. Your installation may also offer a special export or publish skin.
IDEA! The view template is used to generate published pages, so view.basic_publish.tmpl is the template that will be used to generate the output. You can preview any topic in this skin simply by appending ?skin=basic_publish to the end of the view URL. Note that the standard VIEW_TEMPLATE template override still works when publishing (but only if the VIEW_TEMPLATE has some content). 		publishskin
Extra Preferences Lets you define Foswiki preferences that will be available for use in topics during this publishing run. Define preferences one per line, using the syntax PREFERENCE=VALUE - for example, 
TOOL_VERSION=3.14.15
ISBN=1-56592-149-6
Preferences defined this way can be used in topics (including the history topic) like any other Foswiki preference. 		preferences
Enable/Disable Plugins
Comma-separated list of plugins to enable during publishing.

You can enable a normally-disabled plugin by giving the plugin name e.g. MyPlugin, or disable a normally-enabled plugin by prefixing it with a minus sign e.g. -CommentPlugin. You can disable all plugins by starting the list with -* and then selectively enable plugins again later in the list e.g. -*, SmiliesPlugin, AutoViewTemplatePlugin, HomePagePlugin, InterwikiPlugin.

You are recommended to disable any plugins that generate interactive buttons in the output. Only plugins present in configure can be enabled/disabled. 		enableplugins
Template By default the plugin uses the default view template when it renders topics for publishing. You can override this here. Note that this is a very specialised feature; you will normally only need to override the skin when publishing. template
Copy External Resources Copy externally referenced resources (e.g. images on other servers). This option enables the copying of resources hosted on external servers into the published content. If it is disabled, the plugin will maintain an internet link to the external content. Enable this option if you want the pubished content to be totally self-contained (for example, for offline reading) or disable it for faster publishing process and smaller output. copyexternal
Publish All Attachments Normally only attachments that are explicitly referenced in the text are published. Enable this option to publish attachments on topics that are not referenced in the text as well. allattachments
Publishing history topic This is where the history of your publishing is stored. Specify a full web.topic name (you must have write access). Each time you publish, this topic is re-written with the log of the publishing process. You need CHANGE access to this topic. You can leave this blank if you don't need a history. history
Debug Enable to get a lot of debug messages, mainly relating to the processing of URL. debug
Output options
The plugin can use a number of different 'back ends' to generate output in different formats. These are selected using the format parameter. Open the relevant tab below to select a format and set parameters.

Using a rest call

Create a link that invokes the rest script and pass the current topic: (added newlines for readability). 
<a class='foswikiPopUp'
href='%SCRIPTURLPATH{"rest"}%/PublishPlugin/publish?%REVARG%;
topics=%BASEWEB%.%BASETOPIC%;
format=file;
rel='nofollow'>
Publish this page
</a>

Using Bookmaker

The Bookmaker allows you to select a number of topics by visiting them in turn and adding them to the book. Once your book is complete, you can return to this page to publish it.

To start the bookmaker, enable the BookmakerPlugin, and revisit this page to enable the interface. Once the bookmaker is running, visit the pages you want to add to the book and add them to the book. Once you are finished, use the bookmaker interface to return to this page to publish the results. To publish a book generated by Bookmaker, use the %BOOKLIST{"Bookweb.BookName"}% macro in the topics parameter.

Using a Publish Topic (configtopic)

You can create a publish topic that contains all the details needed to publish. This is just a topic with a series of standard preference settings (which correspond to the parameters described here) in it.

You can use the PublishWeb topic in this web as a template for your own topics.

To use a publish topic, you must pass the configtopic parameter to the publish script set to the name of the topic to use to control publishing. This should be a full web.topic specification (if you only give a topic name it will pick the topic up from the Main).

Publishing from the command line

This is the recommended way to publish if you are regularly updating published content. Just cd to the bin directory, and perl rest /PublishPlugin/publish. Parameters are passed as name=value pairs, for example: 
perl rest /PublishPlugin/publish topics='System.*,-*.Web*' format=file
The available parameter names are shown in the publish form above.

How-tos

How to control which parts of a topic get published

You can control what gets published from a topic using %STARTPUBLISH% and %STOPPUBLISH% control tags:
  • If %STARTPUBLISH% is the first control tag seen in the file, everything before it will be ignored.
  • Everything between %STOPPUBLISH% and the next %STARTPUBLISH% (or the end of the topic) will be ignored.
  • %STARTPUBLISH% and %STOPPUBLISH% will be visible in the viewed topic, so you can easily see what will be published from the topic.
    • If you don't want to see the tags in normal view, then just define global preferences STARTPUBLISH and STOPPUBLISH using Set. Set them to the empty string. That won't stop them being interpreted by the plugin, but will make them invisible in normal view.

Another good trick is to set up a special "publishing" web. Create topics in the web that %INCLUDE the topics from other webs that you want to publish. You can use STARTSECTION and ENDSECTION to highlight what you want published. This way the "publishing" web gives you a view of exactly what will be in the published output, without the need for special publishing tags.

How to use Wildcard Patterns

A wildcard is a special string that you can put into a filename so that it matches a whole range of files:
String What it does Example What the example matches
* Matches any string, including an empty string. *Cheese* Every topic with "Cheese" somewhere in the name (but not "cheese")
? Matches any single character. Example1? Example10 and Example 1X but not example1
[...] Matches any one of the enclosed characters. A pair of characters separated by a hyphen denotes a range expression; any character that sorts between those two characters, inclusive, using the current locale's collating sequence and character set, is matched. If the first character following the [ is a ^ then any character not enclosed is matched. A - may be matched by including it as the first or last character in the set. A ] may be matched by including it as the first character in the set.
Within [ and ], character classes can be specified using the syntax [:class:], where class is one of the following classes defined in the POSIX.2 standard: alnum, alpha, ascii, blank, cntrl, digit, graph, lower, print, punct, space, upper, word, xdigit. A character class matches any character belonging to that class. The word character class matches letters, digits, and the character _. 		B[aeiou]g Bag, Bog, Big, Beg, Bug

Irrespective of the archive being used, local output is always generated in the directory specified by the {Plugins}{PublishPlugin}{Dir} configuration setting. Administrators can manage the contents of this directory from the browser using the %PUBLISHERS_CONTROL_CENTRE% macro (see PublishPluginControlCentre).

If relativedir is set, then it will be added after {Plugins}{PublishPlugin}{Dir}. See How to attach the output to a Topic for an example of how to use this.

If outfile is not set in the parameters it defaults to the name of the format being published.

Most formats generate a single file with a unique extension that identifies the format e.g. .pdf. When publishing a format that generates multiple files (e.g. file) then outfile will normally be a directory.

ALERT! The rendered data can get pretty big, and the publishing process itself puts a heavy load on the server, especially when using compression on large webs.

How to generate a Publishing History

Every time a web is published, then the results of that publishing step can be stored in a topic in the web, by setting the history parameter to the name of a topic. In order to publish a web, you have to be able to write to this topic.

If the selected publishing skin defines a skin template called publish_history, then that template will be used as the basis of the history topic. This (for example) allows you to use a template with a skin to define access controls for the history topic. The template can refer to a Foswiki macro %PUBLISHING_HISTORY% to get the expanded history. The basic_publish skin provides templates/publish_history.basic_publish.tmpl for this purpose.

The history topic contains a list of all the parameters used, and the versions of the topics that were published, so it is very useful for tracking exactly what you publish. However it can grow very large, if (for example) you are updating a static site from the wiki content regularly.

How to attach the output to a Topic

If you are using an on-disk file store, such as PlainFile or one of the RCS stores, you can publish an attachment direct to an attachment on a topic.

ALERT! Note that overwriting attachments this way is extremely dangerous, so this should only be done by experts! You have been warned. ALERT!

  • Open configure
  • First set the {Plugins}{PublishPlugin}{Dir} to the same as {PubDir}
  • Then publish with a relativedir setting that corresponds to the attachment directory for the web/topic that you want to attach to
  • If {AutoAttachPubFiles} is enabled, it will automatically be attached to the topic.

Installation Instructions

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.

Open configure, and open the "Extensions" section. "Extensions Operation and Maintenance" Tab -> "Install, Update or Remove extensions" Tab. Click the "Search for Extensions" button. Enter part of the extension name or description and press search. Select the desired extension(s) and click install. If an extension is already installed, it will not show up in the search results.

You can also install from the shell by running the extension installer as the web server user: (Be sure to run as the webserver user, not as root!) 
cd /path/to/foswiki
perl tools/extension_installer <NameOfExtension> install

If you have any problems, or if the extension isn't available in configure, then you can still install manually from the command-line. See https://foswiki.org/Support/ManuallyInstallingExtensions for more help.

IMPORTANT Run configure and complete the installation in the PublishPlugin section.

If you want to generate PDF files, you will need to install a PDF generator, for example htmldoc or prince. Find them using google.

Note that htmldoc can also be used to generate PostScript. See the htmldoc man pages for details.

If you want zip output you will have to install Archive::Zip.

If you want tgz output, install Archive::Tar.

If you want to use the ftp upload, you will need to install Net::FTP.

WARNING! Anything published is no longer under the control of Foswiki access controls, and if you make the publish output directory visible on the web then you may need to take precautions to prevent accidental leakage of confidential information by restricting web access to this directory, for example in the Apache configuration.

One way to do this is to use the viewfile access rights management to control access to the published content:
  • Create a hidden web named e.g. Publish
  • Make this web public readable (via WebPreferences; ALLOW/DENYTOPICVIEW)
  • Create a topic PublishedContent
  • Change PublishPlugin Settings in configure to publish to pub/Publish/PublishedContent
Access controles on Publish web and !Publish.PublishedContent will then apply.

Dependencies

NameVersionDescription
File::Spec>0Required. Used to analyse URL paths.
File::Copy>0Required. Used to move files around.
File::Path>0Required to manipulate directories.
File::Temp>0Required for temporary files.
LWP>0Optional. Used to include images referenced by absolute URLs
Archive::Zip>=0Optional. Required to generate .zip output
Archive::Tar>=0Optional. Required to generate .tgz output
Net::FTP>0Optional. Required for ftp publishing.
htmldocOptional. Required to generate .pdf output
Digest::MD5>0Optional. Required for fast upload to ftp servers.

Compatibility Notes for version 3.0

  • Only tested on Foswiki 2.0.
  • The compress parameter has been removed.
  • The <nopublish> tag has been removed.
  • The templates parameter has been removed. We couldn't find anyone who was using it. The template parameter provides a subset of it's functionality.
  • Deletion of existing published content before publishing is now under the control of the keep parameter.
  • The web, topiclist, exclusions and inclusions parameters are still supported, but are undocumented and will be removed in a later version. They are ignored if topics is given.

Change History

3.4 (30 Jan 2018) Foswikitask:Item14611: flatdir generator, Foswikitask:Item14612: add support for handling external resources, Foswikitask:Item14613: Foswikitask:Item14613: improve format UI, Foswikitask:Item14513: add HTML to basic_publish skin Foswikitask:Item14614: support parameters passed to resource URLs
3.3 (26 Jan 2018) Foswikitask:Item14540: correct internal resource resolution Foswikitask:Item14610: re-implement defaultpage
3.2 (23 Jan 2018) Foswikitask:Item14609: fix for subweb names
3.1 (23 Apr 2017) Foswikitask:Item14421: Foswikitask:Item14422: bugfix release
3.0 (22 Apr 2017) Crawford Currie extensively rewrote to separate concerns and get rid of some of the poor code that had crept in over the years. Removed ability to publish several templates in one go, due to performance and complexity concerns. Removed flatpdf format, as it's pretty useless with modern htmldoc. Deprecated web, topiclist, inclusions, exclusions and simplified to one parameter, topics, renamed topicsearch to rexclude and skin to publishskin. WARNING: untested on Foswiki <2
2.5 (25 Jan 2017) Update to work with Foswiki 2.x. Foswikitask:Item14198 Foswikitask:Item11808 Foswikitask:Item8898
2.4 (30 Jul 2012) Foswikitask:Item12016 Add capability to construct flat HTML and PDF files. Bugfixes: Foswikitask:Item11988 Foswikitask:Item11339 Foswikitask:Item11345 Foswikitask:Item8260 Foswikitask:Item10597 Foswikitask:Item11346
2.3.2 (10 Aug 2011) Foswikitask:Item10944: Fix publishing of attachments
2.3.1 (14 Jun 2011) Foswikitask:Item10870: support skinning of the history topic. Foswikitask:Item10870: support definition of session preferences from the publish form. Foswikitask:Item10843: bugfix to publishers control centre
2.2.1 (25 May 2011) Foswikitask:Item10578: allow list of publishskins; Foswikitask:Item10580: allow empty outfile param; Foswikitask:Item10585: support all valid topic names in topiclist; Foswikitask:Item10578: made template purging configurable Foswikitask:Item10594: merged patch; Foswikitask:Item10581: correct paths to resources in HTML output. (Diab Jerius and Crawford Currie)
2.2.0 (28 Mar 2011) Foswikitask:Item8225: fix handling of plugin contexts Foswikitask:Item2635: make sure index.html is generated Foswikitask:Item10529: test integration with BookmakerPlugin via new topiclist parameter
2.1.7 (01 Nov 2010) Foswikitask:Item8658: fix rest output when publishing to file, pdf and ftp
2.1.6 (29 Oct 2010) Foswikitask:Item8522: support for Foswiki 1.1.x and also 1.0.x+ZonePlugin Foswikitask:Item1638: fixed finding resources with parameters (e.g. ?t=2365421)
2.1.5 (05 Feb 2010) Documentation update.
2.1.4 (12 Jan 2010) Foswikitask:Item2557: fixed publish head elements added by ADDTOHEAD
2.1.3 (11 Jan 2010) Foswikitask:Item2615: fixed finding resources not inside quotes
2.1.2 (30 May 2009) Foswikitask:Item8168: fixed genopt (extras)
2.1.1 (22 May 2009) Foswikitask:Item8165: fixed missing BASEWEB and other internal preferences. This was resulting in the wrong web being used for some operations
2.1.0 (16 May 2009) Foswikitask:Item1626: fixed META{"formfield" Foswikitask:Item8150: (Marc Schaefer) fix for -skin parameter Foswikitask:Item1557: doc fix Foswikitask:Item1585: allow history topic in different web Foswikitask:Item871: add missing newline at start of topic text. Foswikitask:Item1449: topic publisher was not popping the context correctly Foswikitask:Item1632: improve backporting support
2.0.2 (18 Mar 2009) Foswikitask:Item804: automatically create publish dir Foswikitask:Item8078: support publishing to a subdir of the publish dir, under url param control
2.0.1 (14 Feb 2009) Foswikitask:Item1033: fixed button in PublishWeb
2.0.0 (27 Nov 2008) Foswikitask:Item8019: refactored as a plugin and tested in Foswiki
1.1.0 (7 Jan 2003) Initial version

This add-on started life as the GenHTMLAddon, written by Foswiki:Main/CrawfordCurrie at Motorola. It was then extended by Eric Scouten, and then further fixed and enhanced by Foswiki:Main/CrawfordCurrie (http://c-dot.co.uk). It has also been extended by Foswiki:Main/SvenDowideit and Foswiki:Main/MartinCleaver, and most recently refactored for Foswiki by Foswiki:Main/CrawfordCurrie. Other significant contributors are Foswiki:Main.ArthurClemens and Foswiki:Main.MichaelDaum.

This code is a development of the Architectures and System Platforms group of Motorola Inc. and is protected by the following copyrights:
  • Copyright © 2001 Motorola. All Rights Reserved.
  • Copyright © 2002-2003, Eric Scouten.
  • Copyright © 2004-2006 Crawford Currie http://c-dot.co.uk
  • Copyright © 2006 Martin Cleaver http://www.cleaver.org
  • Copyright © 2006-2017, Foswiki Contributors

The 2005 functionality improvements were sponsored by Wind River Systems

The pdf and tgz output formats were made possible by Sabio Labs

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details, published at http://www.gnu.org/copyleft/gpl.html

This site is powered by FoswikiCopyright © by the contributing authors. All material on this site is the property of the contributing authors.
Ideas, requests, problems regarding AustLII Communities? Send feedback
This website is using cookies. More info. That's Fine