Demonstrator - Reasoning WebAPI issueshttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues2021-05-13T11:47:31Zhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/40Bump version number2021-05-13T11:47:31ZSiebers, MichaelBump version numberChange version number in code and documentationChange version number in code and documentationVersion 1.0https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/39Decide on xxx_time semantics2020-12-04T18:17:58ZSiebers, MichaelDecide on xxx_time semanticsIn `irrelevance_common.pl`, `creation_time/2` works as (+,-), other `xxx_time/2` are implemented as (?,?).
The tests for all of these predicates are commented to test (+,+), (+,-), and (-,+). But actually only (+,+) and (+,-) is tested!In `irrelevance_common.pl`, `creation_time/2` works as (+,-), other `xxx_time/2` are implemented as (?,?).
The tests for all of these predicates are commented to test (+,+), (+,-), and (-,+). But actually only (+,+) and (+,-) is tested!Version 1.0https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/38Harmonize internal and extermal representation of items2020-12-04T19:38:07ZSiebers, MichaelHarmonize internal and extermal representation of items_The described status quo assumes !13 has been merged. Consequently, this issue should only be attacked after said merge._
# Current situation
The external (JSON) representation of files and directories must have the property "type" wit..._The described status quo assumes !13 has been merged. Consequently, this issue should only be attacked after said merge._
# Current situation
The external (JSON) representation of files and directories must have the property "type" with a value of `"file"` or `"directory"`, respectively. The internal (dict) representation does not have this key, but the tag if the dict is either `'file'` or `'directory'`.
When receiving external representations (via `/bg` POST):
- the external representation is read into a dict (where the tag is a variable)
- it is checked if the "type" key is present with a valid value (done before transformation to give meaningful error messages)
- the "type" key is removed from the dict
- the dict's tag is unified with the value of type
When sending external representations (via `/bg` Get):
- the internal representation is retrieved from background knowledge
- it is send as is (This a bug, as we claim that our result conforms to the input specification but we send no "type" property).
# Solution
Either
1. we add the "type" property/key before sending out internal representations,
2. we change our internal representation to include the type in addition to the tag value, or
3. we change our internal representation to include the type but use a uniform tag, for example `item`.Version 1.0https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/37Refactor endpoints2020-12-13T15:13:09ZSiebers, MichaelRefactor endpointsFor naming consistency I propose the following new endpoint names for POST requests:
| New Endpoint | Old Endpoint
|-----|-----|
|`/bg/add` | `/bg` |
|`/bg/remove` | `/clear` |
|`/bg/clear` | `/clear/all` |
The functionally of GET...For naming consistency I propose the following new endpoint names for POST requests:
| New Endpoint | Old Endpoint
|-----|-----|
|`/bg/add` | `/bg` |
|`/bg/remove` | `/clear` |
|`/bg/clear` | `/clear/all` |
The functionally of GET requests to `/bg` should be kept.
# Rationale
All endpoints working on background knowledge work on prefix `/bg`. The suffixes `/add`, `/remove`, and `/clear` convey their meaning more concisely than the original paths. The GET request to `/bg` is clearly separated from other requests.Version 1.0Sebastian SeufertSebastian Seuferthttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/36POST Request with negative Content-Length leads to 500 status2020-12-04T17:46:04ZSiebers, MichaelPOST Request with negative Content-Length leads to 500 statusSending a request with negative `Content-Length` header leads to a `500 Internal Server Error`. Error message is accompanied by an HTML error message.
This should be a `4xx` status with a JSON error message.Sending a request with negative `Content-Length` header leads to a `500 Internal Server Error`. Error message is accompanied by an HTML error message.
This should be a `4xx` status with a JSON error message.https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/35Undocumented edge case for adding background knowledge2020-12-02T17:01:10ZSiebers, MichaelUndocumented edge case for adding background knowledge# Summary
Adding two items with the same absolute path succeeds if some other property is distinct.
# Steps to reproduce
Send a POST request to `/bg` requesting to add two items with identical path but at least one distinct property.
E...# Summary
Adding two items with the same absolute path succeeds if some other property is distinct.
# Steps to reproduce
Send a POST request to `/bg` requesting to add two items with identical path but at least one distinct property.
Example cURL call:
```shell
curl --request POST 'http://localhost:4444/bg' \
--header 'Content-Type: application/json' \
--data-raw '[{
"type" : "directory",
"abs_path": "/A/B",
"creation_time": 1
},
{
"type" : "directory",
"abs_path": "/A/B",
"creation_time": 2
}
]'
```
# Observed behavior
The server returns that both items have been added.
```json
{
"added": 2,
"received": 2,
"skipped": 0
}
```
# Expected behavior
Only the first addition should succeed. The second should fail as there is already an item known at this path.Version 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/33Validate JSON input and throw exceptions2020-12-04T17:58:19ZSiebers, MichaelValidate JSON input and throw exceptionsOnly for the `/irrelevant/file` and the `/explain` endpoint, the JSON input is validated. And only these two endpoints throw appropriate exceptions (c.f. !6). These practices should be rolled out to the other endpoints:
- [ ] ~~`/`~~
- ...Only for the `/irrelevant/file` and the `/explain` endpoint, the JSON input is validated. And only these two endpoints throw appropriate exceptions (c.f. !6). These practices should be rolled out to the other endpoints:
- [ ] ~~`/`~~
- [ ] ~~`/add`~~
- [ ] ~~`/remove`~~
- [ ] ~~`/query`~~
- [x] `/state`
- [x] `/bg`
- [x] POST requests (c.f. #21)
- [x] GET requests
- [x] `/clear`
- [x] `/clear/all`
Some of these tasks might be done more easy after #27 is solved.Version 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/32Alternating error responses2020-12-04T17:46:03ZSiebers, MichaelAlternating error responses# Summary
The problem of alternating responses as expected and error responses, like described in #17 and #18, still persists. This happens if the HTTP stream is kept open (keep-alive), no `Content-Length` header is sent, the request bod...# Summary
The problem of alternating responses as expected and error responses, like described in #17 and #18, still persists. This happens if the HTTP stream is kept open (keep-alive), no `Content-Length` header is sent, the request body contains a valid payload, and there is more data after the valid payload.
# Steps to reproduce
Taking the `/irrelevant/file`-endpoint as example:
1. Send a POST request to endpoint `/irrelevant/file` ...
- with body `{"abs_path":"a"}_`,
- with header `Content-Type: application/json`,
- with header `Connection: keep-alive`,
- without header `Content-Length`
2. Send any request to any endpoint (may be identical)
cURL command to reproduce
```curl
curl --request POST 'http://localhost:4444/irrelevant/file' \
--header 'Content-Type: application/json' \
--header 'Content-Length:' \
--data-raw '{"abs_path":"a"}_' \
--next --request GET 'http://localhost:4444/bg'
```
## Observed behavior
- first request succeeds with `{"result":true}`
- second request fails with a HTML-formatted `400 Bad Request` message stating `Illegal HTTP request`
## Desired behavior
I'm not sure what this should be. Depending on the additional payload supplied, the first request could be a `400 Bad Request`. If it's only white space the request may be considered ok. But if it is _real_ garbage it should be dismissed.
Either way, the second request should succeed (if it is correctly formatted)
# Solution
Internally, the predicate `http_client:http_convert_data/4` (defined in library http/http_json) is used to parse the JSON payload. The behavior of this predicate depends on the `Content-Length` header:
- If it is not supplied, the stream is read (and parsed) until parsing the first JSON object succeeds. Further data is left on the stream.
- If such a header is supplied, the stream is read (and parsed) until the full content length is read or parsing the first JSON object succeeds. Further data (up to the content's length) is discarded.
The first case leads to the described problem, as additional payload will "start" the next request making it invalid. The second case is also problematic itself, but this is not relevant for this issue.
There are different ways to circumvent the problem:
1. Disallow POST requests without `Content-Length` header.
1. Always close the stream after each request (disregarding the `Connection` header)
1. After parsing the JSON payload, read the stream until the end and manually handle excess payload.Version 1.0Sebastian SeufertSebastian Seuferthttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/31Document allowed characterpatterns for abs_path2020-12-10T18:07:35ZSebastian SeufertDocument allowed characterpatterns for abs_pathEither as schema or as regular expression.
@michael.siebers since this is related to your typechecking, where to best document this?Either as schema or as regular expression.
@michael.siebers since this is related to your typechecking, where to best document this?Version 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/30'Request zero' at GET2020-11-30T16:09:54ZSebastian Seufert'Request zero' at GETAPI reacts with completed request 0.
```server(started, 1606395900).
/*Thu Nov 26 14:04:59 2020*/
request(1, 1606395899.985, [peer(ip(127,0,0,1)),method(get),request_uri('/bg'),path('/bg'),http_version(1-1),content_type('application/js...API reacts with completed request 0.
```server(started, 1606395900).
/*Thu Nov 26 14:04:59 2020*/
request(1, 1606395899.985, [peer(ip(127,0,0,1)),method(get),request_uri('/bg'),path('/bg'),http_version(1-1),content_type('application/json'),user_agent('PostmanRuntime/7.26.8'),postman_token('fc04c4f5-3f71-40a2-b3fe-66fc358a9f10'),host(localhost),port(5000),content_length(735)]).
completed(1, 0.029408105, 630, 200, ok).
completed(0, 0.047579236, 328, 400, error("Illegal HTTP parameter: { (in_http_request)")).
server(stopped, 1606395987).
```
----
I have no idea at the moment what causes this / where this comes from, but since it does not crop up in Postman or curl, I assume this _should_ not interfere with API functionality from the clients perspective.
Investigating.Sebastian SeufertSebastian Seuferthttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/29/clear endpoint not working2020-12-04T17:46:04ZSyed Mamoon Ahmed/clear endpoint not workingMy request is:
> {
> "type" : "file",
> "abs_path": "/A/B/Filename-2.exe",
> "file_size": 1024,
> "creation_time": "1604235569",
> "modification_time": "1604235569",
> "access_time": "1604235569",
> "change_time": "1...My request is:
> {
> "type" : "file",
> "abs_path": "/A/B/Filename-2.exe",
> "file_size": 1024,
> "creation_time": "1604235569",
> "modification_time": "1604235569",
> "access_time": "1604235569",
> "change_time": "1604235569",
> "media_type": "img",
> "filename_extension": "png"
> }
The response is this:
<> html>
> <head>
> <meta content="HTML Tidy for Java (vers. 26 Sep 2004), see www.w3.org" name="generator"/>
> <title>500 Internal server error</title>
> <meta content="text/html; charset=UTF-8" http-equiv="content-type"/>
> </head>
> <body>
> <h1>Internal server error</h1>
> <p>
> Arguments are not sufficiently instantiated
> <br/>
> In:
> <br/>
> [20] _6592 is _6598-1
> <br/>
> [19] web_api:clear_bg([protocol(http),...|...]) at /home/transmatter/folders/code/uniBam/thesis/demonstrator/web_api.pl:266
> <br/>
> [18] http_dispatch:call_action(web_api:clear_bg,[protocol(http),...|...]) at /usr/lib/swi-prolog/library/http/http_dispatch.pl:1030
> <br/>
> [16] time:run_alarm_goal('$alarm'(11197176410828),http_dispatch:call_action(...,...,...)) at /usr/lib/swi-prolog/library/time.pl:147
> <br/>
> [15] setup_call_catcher_cleanup(time:alarm(300,...,...,...),time:run_alarm_goal(...,...),_850,time:remove_alarm_notrace(...)) at /usr/lib/swi-prolog/boot/init.pl:564
> <br/>
> [8] httpd_wrapper:call_handler(web_api:http_dispatch,28,[protocol(http),...|...]) at /usr/lib/swi-prolog/library/http/http_wrapper.pl:329
> <br/>
> [7] catch(httpd_wrapper:call_handler(...,28,...),error(instantiation_error,context(...,_1000)),httpd_wrapper:true) at /usr/lib/swi-prolog/boot/init.pl:482
> <br/>
> [6] httpd_wrapper:handler_with_output_to(web_api:http_dispatch,28,[protocol(http),...|...],current_output,error(instantiation_error,context(...,_1078))) at /usr/lib/swi-prolog/library/http/http_wrapper.pl:306
> <br/>
> [5] httpd_wrapper:handler_with_output_to(web_api:http_dispatch,28,[protocol(http),...|...],<stream>(0x7f071c03dd50),error(instantiation_error,context(...,_1150))) at /usr/lib/swi-prolog/library/http/http_wrapper.pl:318
> <br/>
> <br/>
> Note: some frames are missing due to last-call optimization.
> <br/>
> Re-run your program in debug mode (:- debug.) to get more detail.
> </p>
> <address>
> <a href="http://www.swi-prolog.org">SWI-Prolog</a>
> httpd at transmatter-laptop
> </address>
> </body>
> </html>
and the log is attached:
[log.txt](/uploads/6c3cc7d695f0c3347e4915d18510b856/log.txt)
PS. There was some documentation on how to use these endpoints in README.md. I cannot find it anymore, so I just used my previous requests.https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/28Force errors to be JSON2021-06-08T15:42:32ZSiebers, MichaelForce errors to be JSONThe http server library decides automatically if errors should be returned as formatted HTML or as JSON. This decision is made in `http_json:prefer_json/1`.
We always want JSON returns. However, we *may* get HTML returns, for example if...The http server library decides automatically if errors should be returned as formatted HTML or as JSON. This decision is made in `http_json:prefer_json/1`.
We always want JSON returns. However, we *may* get HTML returns, for example if the request does not contain an `Accept` header. Can we force the library to return JSON?Version 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/27Add regex types to types module2020-12-01T01:41:16ZSiebers, MichaelAdd regex types to types module# Current state
In the types module the type of objects can be checked. This is not restricted to Prolog's definition of types: at least for dicts required and optional keys (with type restrictions) may be defined.
# Suggestion
The rege...# Current state
In the types module the type of objects can be checked. This is not restricted to Prolog's definition of types: at least for dicts required and optional keys (with type restrictions) may be defined.
# Suggestion
The regex type would extend this idea to strings: the value must not only be of type string but its content must satisfy additional requirements. Additionally, a regex type for `atom`s or general `text` is desired.
## Rationale
We have decided, that absolute paths of files and directories may not end in a slash unless it is the root directory '/'. Without the regex type, this restriction must be enforced after type checking. However for this, a mechanisms very similar to type checking are required. For example in `set_bg`:
- if we have a single dict:
- noting which values are concerned (probably only `abs_path`),
- extracting the value
- *checking the regex*
- if the check fails: throw exception
- if we have a list of dicts:
- iterate over all elements
- if we then have a dict, apply the steps from above
It is easier, less error prone, and more encapsulated to include this facility into types.
## Technical realisation
The actual checking of the type might be simple as there is a regex library in Prolog. Taking strings as example, the type definition could look like
- `regex(string, RegEx)` where `RegEx` must be a valid regular expression from Prolog's regex library or
- `string(RegEx)` where `RegEx`is defined as above.
The second option would convey the notion that this is a refinement of the string type more strongly. This is analogous to `dict` vs. `dict/1` and `dict/2` and `list` vs `list/1` already realized in `types`.https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/26Startup parameters2020-11-26T13:39:06ZSiebers, MichaelStartup parametersStartup options for the server (like port to run on) seem to be managed in the predicate server_opts/1. However, I do not use how to use them. Regarding the port option, for instance, neither of the following works:
```
swipl -s run.pl ...Startup options for the server (like port to run on) seem to be managed in the predicate server_opts/1. However, I do not use how to use them. Regarding the port option, for instance, neither of the following works:
```
swipl -s run.pl --port 8976
```
```
swipl -s run.pl --port=8976
```
```
swipl -s run.pl port=8976
```
According to the swipl usage message, swi-options and programm arguments should (can?) be separated by `--`. This seems to kinda work. However, then our code complains that `--` is no valid argument.
Basically, I have the following questions:
1. How is it possible to start the server with arguments?
1. Which arguments are defined? Or rather, what do they do?
1. Can we print some usage message ourselves? For example with `-h` but without interfering with swipl's `-h` argument?Version 1.0Sebastian SeufertSebastian Seuferthttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/25Describe project2020-12-12T19:06:59ZSiebers, MichaelDescribe projectGive the GitLab project a telling name, maybe provide additional information / linksGive the GitLab project a telling name, maybe provide additional information / linksVersion 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/24Document theory_common and theory_templates2020-12-04T19:40:05ZSiebers, MichaelDocument theory_common and theory_templatesDescribe ...
- [x] the idea behind predicates in `theory_common.pl`
- [x] where these predicates are (or might be) used
- [x] the format of explanation templates
- [x] how to load explanation templates
- [x] where explanation templates a...Describe ...
- [x] the idea behind predicates in `theory_common.pl`
- [x] where these predicates are (or might be) used
- [x] the format of explanation templates
- [x] how to load explanation templates
- [x] where explanation templates are usedVersion 1.0https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/23Automate API-tests2021-06-12T09:00:31ZSiebers, MichaelAutomate API-testsAPI testing could be automated using Postman. There test requests with success conditions can be defined. These requests can be stored in collections which may be executed from CLI/CI.API testing could be automated using Postman. There test requests with success conditions can be defined. These requests can be stored in collections which may be executed from CLI/CI.https://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/22Map http_reply exceptions to JSON reply2020-12-01T12:37:34ZSiebers, MichaelMap http_reply exceptions to JSON replyVersion 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/21Add key and type check for file/directory JSON payloads2020-12-04T17:46:03ZSiebers, MichaelAdd key and type check for file/directory JSON payloadsSeveral endpoints require information on files or directories as JSON objects. When parsing these objects we should check that
- no required key is missing,
- values are of appropriate / expected type and
- no unknown key is provided.
W...Several endpoints require information on files or directories as JSON objects. When parsing these objects we should check that
- no required key is missing,
- values are of appropriate / expected type and
- no unknown key is provided.
Without these checks _semantic_ errors may occur which are hard to debug / detect (c.f. #19).Version 1.0Siebers, MichaelSiebers, Michaelhttps://gitlab.rz.uni-bamberg.de/cogsys/dare2del/demonstrator/-/issues/20/explain: Is it supposed to be like this?2020-11-23T15:35:40ZSyed Mamoon Ahmed/explain: Is it supposed to be like this?Please tell me what am I doing wrong?
Here is my request:
> {
> "abs_path": "/A/B/Filename-2.exe"
> }
Here is the response:
> HTTP/1.1 400 Bad Request
> Date: Mon, 16 Nov 2020 21:43:23 GMT
> Connection: close
> Content-Le...Please tell me what am I doing wrong?
Here is my request:
> {
> "abs_path": "/A/B/Filename-2.exe"
> }
Here is the response:
> HTTP/1.1 400 Bad Request
> Date: Mon, 16 Nov 2020 21:43:23 GMT
> Connection: close
> Content-Length: 350
> Content-Type: text/html; charset=UTF-8
>
> <!DOCTYPE html>
> <html>
> <head>
> <title>400 Bad Request</title>
>
> <meta http-equiv="content-type" content="text/html; charset=UTF-8">
>
> </head>
> <body>
>
> <h1>Bad Request</h1>
>
> <p>
> Unknown message: not_irrelevant('/A/B/Filename-2.exe')</p>
>
> <address>
> <a href="http://www.swi-prolog.org">SWI-Prolog</a> httpd at transmatter-laptop
> </address>
>
> </body>
> </html>Sebastian SeufertSebastian Seufert