More fun with curl - Web forms and access rights (you will not like this)

We had fun with curl and LotusLive before. Now lets have some fun with curl and web forms. To follow the fun you should download and install cURL (for Ubuntu users: sudo apt-get install curl). When discussing web enablement and XPages the topic of "field validation" pops up quite regular. My advice here stands firm: " client side validation solely serves the comfort of the users (which is important) but never the integrity of your data". More often than not I'm greeted with disbelieve. However at the very end what is send back to the server is just a HTTP post and the server developer has little control how that post was generated. In a modern browser one can use Firebug or GreaseMonkey to alter the upload. All browsers can be redirected to send data via a local proxy (try TCPMon for fun or any other local proxy). While that might be out of the reach of NMU ^*, a simple cURL command is not. I will use Domino as backend for my examples, however the approach applies to all type of backends: ASP, PHP, JSP, JSF, Ruby, NodeJS and whatever. Some of the considerations are Domino specific (around back-end behavior).
To send (HTTP POST) data to a server one can simply use curl -F 'fieldname=fieldvalue' . However that command line can get messy when there are a lot of fields and eventually authentication and redirections. So I will a) use a file as source for upload b) use the following command: curl --netrc --data @filename --post301 --post302 . The parameters post301, post302 make sure curl follows redirections if they occur without dropping the POST method, --data @ uses a filename as source for the upload, finally --netrc tells curl to look in a secure (mode 600) file .nerc ( _netrc on Windows) for a line: machine hostname login username password password (replace the bold parts with your information) if authentication is required. You can wrap this into a shell script to reduce typing. I created a simple form Fruit with 3 fields: Subject, Color, Taste and one view Fruits sorted by subject in my curldemo.nsf. Let the fun begin:

1. • Test: Post Subject=Durian&Color=White&Taste=don't ask!!! to http://localhost/curldemo.nsf/fruit?CreateDocument
     In this case an authorized user posts exactly the fields that are in the HTML form to the designated address.
   • Result/Insight: Works as expected. You get a message back (with some html decoration) Form processed.
2. • Test: Post Subject=Apple&Color=Green&Taste=sour to "http://localhost/curldemo.nsf/fruit?OpenForm&Seq=1"
     One could expect the same result as in the first test, since it is all fields, a valid url and an authorized user.
   • Result/Insight: Domino comes back with an empty form and no document gets created. On closer inspection you will find an extra hidden field "__Click" in the form Domino provides. So when you change your post data to Subject=Apple&Color=Green&Taste=sour&__Click=0 the document gets created as expected.
3. • Test: Remove your user's right to create documents by demoting to author with "Create documents" unchecked. Post Subject=Apple&Color=Green&Taste=sour to http://localhost/curldemo.nsf/fruit?CreateDocument
   • Result/Insight: As expected you get an HTTP Error 401 (not authorized) message
4. • Test: Promote your user to Editor and post Subject=Apple2&Color=Green And Green&Taste=deadly&__Click=0 to http://localhost/curldemo.nsf/Fruits/Apple2?EditDocument&Seq=1
   • Result/Insight: The document gets saved back to the server and you get a "Form processed" message back. Works as expected. If you use a fruit name that doesn't exist you get a 404 - Entry not found in index error. You can change the key field when you post Subject=Deadly Apple&Color=Green And Green&Taste=deadly&__Click=0
5. • Test: Update the input validation formula of the Color field to @If(@Trim(@ThisValue)="";@Failure("Please submit a color");@Success). Post Subject=JackFruit to http://localhost/curldemo.nsf/Fruit?CreateDocument In this example we only post a subset of the fields to create a document failing the server side validation
   • Result/Insight: As expected the creation of the document fails and an error is returned.
6. • Test: Post Color=Red And Green&__Click=0 to http://localhost/curldemo.nsf/Fruits/Deadly%20Apple?EditDocument&Seq=1 In this example we only post a subset of the fields back to the form
   • Result/Insight: The existing fields in the document are preserved and only the submitted values are updated. Works as expected.
   
   So far we were playing nice and got expected results. Now let's get sneaky.
   
   

7. • Test: Post Color=Blue&Form=SuperFruit&__Click=0 to http://localhost/curldemo.nsf/Deadly%20Apple?EditDocument&Seq=1 Form does exist as item in the document, but not as field in the form.
   • Result/Insight: The color field is updated while the form field is silently dropped. This also holds true for other field names that are not in the form. In older versions of Domino submitting a field that didn't exist resulted in a HTTP error 500. So the Domino server is more forgiving what you can send, but still quite strict what it will process
8. • Test: Until now all fields we were posting to were visible html fields. A really interesting question however is: what happens to fields that are not rendered as visible html input fields: Fields with the attribute type=hidden, input fields marked as "hide from browser" (or any other hide-when formula), visible computed fields, hidden computed fields (both with the special case of having @ThisValue as formula, visible computed when composed fields, hidden computed when composed fields, visible computed for display fields and last not least hidden computed for display fields (an Oxymoron that only Domino developers can appreciate).
   • Result/Insight: The table below shows the results: Field type Visible Hidden by @HideWhen Standard input Note item gets updated Note item gets updated Not in Form posted value discarded posted value discarded Input type="hidden" Note item gets updated Note item gets updated Computed @ThisValue Note item gets updated Note item gets updated Computed posted value replaced by formula result posted value replaced by formula result Computed when composed Note item gets updated Note item gets updated Computed for display posted value discarded posted value discarded
   The results have quite some ramifications for your average workflow application! A typlical layout would be to use hide-when formulas depending on the role of the user and the current workflow status. Typically the submitter continues to have author access to a document, since (s)he can cancel the request until (or even after) final approval. The web form might not allow any changes to submitted values, but with a little curl that form is easily updated. In consequence: a secure web application must assign a value to the form field that only contains fields that should be editable by a given user and additional: the number of users in Author fields should be limited at all times to the bare minimum. Of course your webquerysave agent can check who is submitting and should (s)he be allowed to do that. To be fully on the safe site you can use the inversion of logging pattern.

There is plenty of good information about curl on superuser.com, details in the curl manual, or the curl scripting guide (a better annotated version is here). Depending on your application server and the "defenders" efforts you might need to deal with headers, cookies or referrers, all of this can be dealt with.
As usual: YMMV

^*NMU = Normal Mortal User