Just modified the js in /httpd/iw/tinymce/jscripts/tiny_mce/plugins/iwimg/* but it doesn't work.
Share your attempted changes to this file and maybe we can see where you went wrong. It would also help if you clarified "doesn't work".Yes, this can be done, though it would be safer (read: better supported) for you to copy the iwimg plugin and modify the copy, then replace the original plugin with yours in the toolbar.
var browseWin = opener.top._dcWin.ShowIMGSrcBrowser(window, "setImage", imgpath);