Workflow Modelling

In the previous exercises, you have learnt about the OGC WPS specification and how to implement a geoprocessing workflow (GPW) using the concept of process chaining. Some of the exercises required you to perform programming work to create and execute geoprocessing functions. For some of you with little programming skills, you must have learnt a lot of new skills though with much effort. In this exercise, you are going to do little programming but use a workflow management system (WfMS) that offers an abstraction of the difficult tasks you have carried out.

A workflow management system defines, manages and executes workflows through the execution of software whose order of execution is driven by a computer representation of the workflow logic. As shown in Figure 1, a WfMS is made up of two main components, the workflow client and the workflow engine. The workflow client contains a graphical editor which allows users to interactively compose the visual workflow by dragging and dropping the figures representing the workflow elements. It also allows users to define the rules and sequence of execution of the workflow. Additionally, the workflow client has a monitor where users can view the result of their workflow once execution is completed. After visually modelling a workflow, it is translated to a script which is sent to the workflow engine where execution takes place. An example of a workflow engine that you have used in the previous exercise is the wps.py file. This file was responsible for controlling the execution of the processes which you defined.

Workflow modelling involves using a tool to visually define the sequence of processes and decisions that have to be carried out to solve a particular problem. You have learnt in previous lectures that workflow modelling can be achieved using UML, BPMN and other modelling languages. Several software exists today that implement these languages, for instace, the ProcessMaker uses BPMN specification for its workflows. When you create a workflow in ProcessMaker, you can export it as a BPMN document and reuse it in other WfMS that implement BPMN. This therefore support sharing and reuse of workflows. A BPMN document is an XML based file representation of the graphical workflow. Other WfMS that do not implement BPMN uses their own standards to specify workflows. For instance, ILWIS, ERDAS and QGIS 2 uses a JSON file format which implements their own specification.

This exercise is going to introduce you to a workflow application (WorkflowApp) that has adopted some of the standards used by other WfMS to specify workflows. We are going to demostrate to you how you compose workflows using your own web services or services from different providers. After composing your workflow, you will be able to submit it to a workflow engine which is responsible for the execution. The result of your workflow's execution will be sent back to you and visualized in the workflow client.

The WorkflowApp

Open the following URL in a new tab of your browser and wait until all the components have loaded. When fully loaded (Figure 2), you will notice that the application has got three main sections; the web services panel, workflow editor panel and the viewer/result panel. By default, the application loads with a default settings for processes and data services. A workflow used in the AfriAlliance (AA) project is also loaded by default.

https://gisedu.itc.utwente.nl/exercise/apps/workflow/

The Services Panel

The Services Panel, on the left side of the page, contains a list of web processing services and data services that can be used in the workflow composition. Expand the tree for the processing services to see the list of geoprocessing functions that are offered by different service providers. The geoprocessing functions are implemented in different tools using various languages as you can observe. For instance, the geoprocessing functions of ILWIS (ILWIS4 Operations & ILWIS3 Server) are implemented in C++ whereas the GeoServer functions (Default Server) are implemented using Java. As you learnt in the lectures, web services offers an abstraction layer that hide the implementation details and environment in which these services are implemented. To view the metadata for a process, right-click on a specific process and choose Metadata option. For instance, expand ILWIS3 Server (WPS) to view list of available geoprocessing functions. Select the Moving Average and right-click to view metadata information.

The data service panel provides a list of data shared using WCS, WFS and SOS. WCS has been used in this application to retrieve raster data(coverage), WFS (vector) and SOS for sensor data.

The Editor Panel

You will use the editor panel to compose your workflows and submit them for execution. The panel allows you upload/download, edit and delete a workflow. The Editor Panel has two tabs, Workflow and JSON. The Workflow tab contains the visual representation of the workflow whereas the JSON tab contains the JSON text representation of the workflow. The toolbar of the Editor panel contains buttons which can be used to achieve a particular purpose. The Execute button ( ) initiates the execution of the workflow. The Delete button ( ) will clear the workflow components from the editor. The drop-down menu contains additional menu items for handling further workflow functions such as importing existing workflow into the editor, exporting a workflow from the editor to a local folder, selecting predefined workflows, adding your own custom processing/data services and updating the name of a workflow in the editor.

The Result/Viewer Panel

This panel allows you to see the result of your workflow execution. Additionally, you are able to view the input data by dragging it from the data services panel to the view panel. The map in the viewer panel only supports the GeoJSON and GeoTIF data formats. Your vector input/output data should be of the Mercator projection (EPSG:3857) since it is what OpenLayers uses. Otherwise, a different coordinate system/projection will take you out of your world.

To view data from the list of data services, select your layer of interest, drag and drop it to the map in the view panel. For instance, select the aa:boundary from the list of data services under Default Server (WFS). Drag and drop it to the map in the viewer panel. The layer as you can observe, Figure 5, is a boundary map for Dano which is one of the study areas for the AfriAlliance project.

Workflow Composition

Now that you are familiar with the application, we are going to visually compose the workflow we used in the previous exercise for the OGC process chaining. Please refer to Figure 1 in section Process Chaining for the abstract workflow and the workflow requirements. Navigate to the page for the WorkflowApp and refresh the web page to reload all the web services for you. Click the delete button and accept to clear the current workflow from the editor. Now you should have a clean editor.

At this moment DO NOT refresh your page or else you will lose all the changes you have made. The application currently does not store your sessions in the browser.

Go to the processing services panel and expand the Default server (WPS) tree. When you scroll down, you will find a list of services offered in this WPS server. From the abstract workflow in Figure 1 in section Process Chaining, we require two geoprocessing functions which are the intersection and a function to calculate the length of a linestring. Identify the similar name of functions from the list of processes and view their metadata information. If you identified correctly, the functions are gs:Intersection of Feature Collections and gs:Length as shown in Figure 6. We will add the functions to the editor panel in the same order as specified in the abstract workflow.

Please add the two functions to the editor. This can be done either by dragging and dropping the functions to the editor or right-clicking on the function and selecting "Add to Workflow". Once you have added the two functions to the editor, the next step will require you to link them. The output of the first function should go to the correct input of the second function. To link the functions, click on the lower blue circle from the source function. While holding the mouse, drag the pointer and drop it on the top circle of the target function. When you release the mouse, a small window will pop up where you will map the outgoing parameter (output from the source function) to the incoming parameter (input to the target function). Once you have correctly mapped the two processes, click the apply button. You will notice that the text color for the target input changes from black to green which shows that the input has been assigned.

The intersection function requires a minimum of two inputs. In the previous exercise, we used the streets and neighbourhood data as inputs. We are going to use the same data sets in this example. Click on the header part of the operation box for the intersection function and select the property icon. This will open up a window where you can fill in the input data requirements for your function. Please only fill the required parameters. Copy the URLs below to the first and second input parameters as shown in the Figure 8 below. NOTE that we are now using EPSG:3857 as the srsname instead of EPSG:28992. This is because we want to view the result in the map. Once you have filled in the required input parameters, click the Save button. The corresponding texts for the inputs in the process box will have the text color of green.

https://gisedu.itc.utwente.nl/cgi-bin/mapserv.exe?map=d:/iishome/exercise/data/afrialiance/layers.map&version=2.0.0&service=WFS&request=GetFeature&typeName=neighbourhood&outputFormat=geojson&srsname=EPSG:3857&buname=Boddenkamp

https://gisedu.itc.utwente.nl/cgi-bin/mapserv.exe?map=d:/iishome/exercise/data/afrialiance/layers.map&version=2.0.0&service=WFS&request=GetFeature&typeName=streets&outputFormat=geojson&srsname=EPSG:3857

Finally, we will assign a name to our workflow. Go to the text box near the Configuration button and type a suitable name for yourself. Thereafter click the Update button to save your changes.

Workflow Execution

After composing the workflow, we are set for testing. Click the Execute () button in the editor's toolbar. This action will submit your workflow to the workflow engine for execution. This process should take a few moments depending on your workflow specification. You will be notified when the execution is successful. The intermediary and final outputs will be displayed in the viewer panel. For a spatial output, the result will be shown in the map. The spatial output can be downloaded by clicking the download button in the Viewer panel. For a non-spatial output, the result will be shown as a text as in Figure 6-9.

Sharing your Workflow

Now that you have a working workflow, you can download it and use it later or also share it with other users or WfMS. Navigate to the JSON tab in your Editor panel. You should now see the textual representation of the workflow in a JSON format. Study the schema used in the JSON format. From this schema, it is possible to reproduce such a workflow and execute it. Download the workflow by selecting the Export your workflow option in the toolbar's menu. Share the file of the workflow with your fellow students. You can reuse the workflow from your fellow student by uploading it to the editor and executing it with your own data. Choose the Import a workflow menu option from the toolbar and select the file of the workflow that has been shared by your fellow student.

Using your Services in the WorkflowAp

You are now able to create geoprocessing functions and execute them with data. When you share your geoprocessing function with your colleagues, you acts as a service provider offering processing as a service. Service providers who implement their services using standards such as the OGC processing and data services make it possible to achieve interoperability and accessibility of geoprocessing resources. For instance, you have used some geoprocessing functions in the exercises which you did not implement. Those geoprocessing functions were based on the OGC WPS thus making it easy for anyone who understands the OGC specifications to consume them.

To distribute your services, you provide the users with the link to the server hosting your services. In the previous exercises we used geoprocessing functions from a GeoServer WPS extension hosted in the URL below. We used this URL to make WPS requests to the server such as the GetCapabilities, DescribeProcess and Execute.

https://mara.rangelands.itc.utwente.nl/geoserver/ows?

You also created your WPS geoprocessing functions which you can also share using the URL below.

https://gisedu.itc.utwente.nl/student/<<SNUMBER>>/gpw/wps.py?

The workflowApp gives you the ability to use web services from different service providers. We are going to show you how you can add your geoprocessing functions to the WorkflowApp and use them to compose workflows.

Open the following URL in a new tab of your browser and wait until all the components have loaded. When fully loaded, clear the editor panel by deleting the AfriAlliance workflow.

https://gisedu.itc.utwente.nl/exercise/apps/workflow/

From the drop-down menu, select the Locate the Add web services option. A new window will pop up as shown in the Figure 10 below. You can see a list of processing and data service providers some of which their services we have used in the previous exercises.

Click the Add button in the Processing services section. A window for adding Processing Service will pop up. Under Resource, select WPS. Enter the name of your WPS server, e.g your <<SNUMBER>>. Enter the URL for your WPS server as shown in Figure 11 and click the Save button. You will get notified that WPS processes have been added to your list of processing services. Close the pop up window and the the configuration panel.

Now if you go to the services panel, you will find your web processing services listed under the name of the WPS server you provided. View the metadata of the processes in your WPS list to see if they are the right ones. Ask your colleague to share the link to their WPS server and then add it to your list of processes. Use some of your processes together with the ones from different service providers available in the list to compose and execute simple workflow of your own.

Do you have a URL for some data service which is OGC WFS/WCS/SOS compliant? If so, use the Add new services menu option to add the data service provider to your list of data services.