suggest you review how the following class works and extend it to preform its tests plus your custom tests
com.documentum.web.formext.action.LaunchComponentWithPermitCheck
once done you need to update the action definition for checkout to use your new class
You can use a custom container that can 1) call canCommitChanges() on CheckoutComponent 2) call onCommitChanges() only for docs that returned true 3) keep list of the ones that couldnt be checked out 4) display the list of documents that could not be checked out at the end.