Exposing Talend Job Metadata from webservice

Background :Deploying TOS jobs as Axis webservices is a great solution to remotely and easily run processes.
When there is the need of returning a data flow from the service, an XML output is returned to the client.This output can be then consumed by the (caller) client application.
Unfortunately the generated XML does not contain metadata information, which is probably ok for some purposes, but we needed it, so we developed a solution, which we are sharing with you.

The solution

A single JSP file (getMetadata.jsp) must be placed in the root directory of the war file generated by the TOS "Esport as Axis webservice" functionality.
When Tomcat (or any other compatible web server) installs the war, it also unpacks getMetadata.jsp in the webservice directory (under webapps)
Calling this jsp page will return an xml structure containing the metadata information retrieved from the eventual tBufferOutput component used in the job.The xml format we decided to use is the same used in the schema definition within TOS (meaning you can capture the output of this jsp, save it as xml and import it ina aschema definition in TOS)

How does it work

getMetadata does not execute the job to retrieve the metadata, instead it reads the files within the webservice to retrieve this information.
You can check the details in the source code, however in summary, this is the processwe used :
- Read server-config.wsd and look for classname. This is used to identify the finename and directory of the file we need to parse (.item). - Inside the .item file it looks for any tBufferOutput component, if it finds one it reads the metadata information associated to it, reformats and outputs it.

How to use it

Let's assume you have a job called myJob, you export it's version 0.2 as an (war) Axis webservice, this will generate a file called myJob_0.2.war. Download the jsp file here
Using a compression untility, copy getMetadata.jsp in the root folder of the war file (should work also with the zip format).
Deploy your webservice as usual (i.e. copying the war file in the Tomcat webapps filder), it will normally install the jsp together with it.
Assuming your tomcat server responds at http://localhost:8080, you will be able to get eventual metadata information (there will be no metadata if you did not use a tBufferOutput component in your job) at the following address : http://localhost:8080/myJob_0.2/getMedatata.jsp

Feel free to provide us your feedback here

Current version

Version : 0.1
Release Date : April 24 2011




Often, together with this jsp utility, we use a webservice wrapper, also being a jsp.
This webservice wrapper is called instead of calling directly the Talend webservice and will call both the getMetadata and the webservice for you, combining the returned information to produce an xml format which is supposedly easier to handle.
Originally this wrapper was created to feed data returned by the webservice into Qlikview, but it can obviously be used in many different ways.
To use it, simply deploy the wrapper page twswrapper.jsp (check on the bottom of the page if you want to download it), normally in your /webaps/ROOT directory in Tomcat.
Once you have thw wrapped in place, you can call it INSTEAD of the webservice passing a few parameters, those parameters are used in the jsp by the following code :



As you can see the parameters host, service, ver are used to identify the URL of the webservice and from there fetch the metadata using the getMetadata.jsp utility.
Some default values are automatically set (check the code) if host or ver are not passed.
Note : host must contain the port if not the default one (80), something like "localhost:8080".
Then the real webservice is called, however first we check if other additional parameters have been passed (context variables in the Talend job).
Those parameters ar identified by the wrapper by checking if they start with the symbol "$".



So, imagine you have a job called myJob, version 1.0 installed on the host localhost:8080, to wich you want to pass the context parameter city=London, you will have to use the following URL :
http://www.localhost:8080/twswrapper.jsp?host=localhost:8080&service=myJob&ver=1.0&$city=London

The wrapper will automatically call the metadata utility and the webservice for you, reformatting the resulting xcml code.
Note that the metadata utility DOES NOT execute the job, it just reads some internal xml files in the webservice package.



Downloads

---

License

THIS SOFTWARE IS PROVIDED BY POWERUP ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL POWERUP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.