Implement crawl state listener

• monitor crawls and update their status on error/complete
• hold state info on queue/progressing internally

(from redmine: issue id 507, created on 2018-01-22tgradl, closed on 2018-01-24)