Online knowledge validation with prudence analysis in a document management application

Prudence analysis (PA) is a relatively new, practical and highly innovative approach to solving the problem of brittleness in knowledge based system (KBS) development. PA is essentially an online validation approach where as each situation or case is presented to the KBS for inferencing the result is simultaneously validated. Therefore, instead of the system simply providing a conclusion, it also provides a warning when the validation fails. Previous studies have shown that a modification to multiple classification ripple-down rules (MCRDR) referred to as rated MCRDR (RM) has been able to achieve strong and flexible results in simulated domains with artificial data sets. This paper presents a study into the effectiveness of RM in an eHealth document monitoring and classification domain using human expertise. Additionally, this paper also investigates what affect PA has when the KBS developer relied entirely on the warnings for maintenance. Results indicate that the system is surprisingly robust even when warning accuracy is allowed to drop quite low. This study of a previously little touched area provides a strong indication of the potential for future knowledge based system development.

► We study prudence analysis (PA) as a method for online verification and validation.
► This study investigates the viability of prudence analysis in a real world domain.
► This paper also studies the affect on a KB when the system is trusted by the expert.
► Results indicate the system is robust even when warning accuracy is allowed to drop.
► This study provides a strong indication that PA is a viable approach to V&V.