Eclipse Crashes with 16.2.8 plugin - (We upgraded CS from 16.2.2 to 16.2.7 version)

Hi Experts,

We recently upgaraded our system from 16.2.2 to 16.2.7. When we try to edit our custom module or even out of box module and save/build it eclipse gives Fatal error exception as below.

#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_UNCAUGHT_CXX_EXCEPTION (0xe06d7363) at pc=0x00007ff928b08e6c, pid=3212, tid=0x0000000000001f3c
#
# JRE version: Java(TM) SE Runtime Environment (8.0_192-b12) (build 1.8.0_192-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.192-b12 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [KERNELBASE.dll+0x8e6c]

Any pointers on this will help!

We are using Eclips Mars2 Release 4.5.2,

Regards,

Mayank Gupta

Find more posts tagged with

Comments

Dave Carpeneto

Hi Mayank - often these errors are due to a glitch in the JVM, and upgrading to a newer build gives good results. However given that CSIDE does make native calls (which are also a frwquent cause of this sort of thing): this may be something that we need to investigate here.

If this is of interest to you: feel free to open a ticket with support for this. When opening: it'd be good if you could supply:

the full error file generated by the JVM
information on how reproducible this is
the Content Server logs ( https://knowledge.opentext.com/go/KB3009405 )
the CSIDE logs ( https://knowledge.opentext.com/go/KB12497600 )

... on the CSIDE log stuff: the article is still in the process of getting published. As such here's a recap:

In Eclipse:

Select "Preferences" from the "Window" menu
Select "Oscript" -> "Logging" in the tree on the left side of the dialog window
Ensure that "File Log Level" is set to "Trace"

... once the issue has reproduced: please zip up and the .metadata\.plugins\com.opentext.cside folder (you can find this inside the Eclipse workspace)

Mayank Gupta

Thanks David for you response and helping us out here.

I have created the ticket and uploaded the logs required as suggested by you.

For the JVM part should we try changing the jre that comes with Content Server version (content_server_java_runtime_environment_8u192_win is the one we are using with 16.2.7) and sits in OT root folder or should we try changing the one which eclipse is using form the System ?

Regards,

Mayank Gupta

Dave Carpeneto

Hi Mayank - I just had a look @ the JVM dumps in the ticket. There's two of them, and both state "The crash happened outside the Java Virtual Machine in native code". As such: changing the Java version likely wouldn't change anything (and since these are on the server-side of the CSIDE process: changing the version isn't a good idea, since other things depend on this being as it is).

Further: the two dumps are for different JNI calls (one is getScriptSource(), the other is getFeature()). So given these are different, and given these are both in the C/native code: maybe something is up wit the server itself? Do you see any entries in the Windows event log around the time of issue? Is your system set up to record dumps, and if so are any seen? Does restarting Windows perhaps change the outcome?

Finally:

is the 16.2.2 -> 16.2.7 upgrade the only thing that changed around the time the issue happened?
is this seen for all / most / some build attempts?

Thx, Dave

Mayank Gupta

Hi Dave,

Yes the two dumps would be different as I tried to edit different modules. First it was custom module and second time I tried editing Core module.

The issue occures for every build. I even tried on diffrent machine but same issue occures. I have tried with diffreent plugins as well. The same issue occures with 16.2.6, 16.2.7 and 16.2.8 oclipse plugin.

Even after restart issue pesists. The issue started coming after we did the upgarde of content server from 16.2.2 to 16.2.7. At first we thought it might be due to old plugin and replaced it wih Latest one as per SDK documentation. But the issue still persists.

I could see below two errors in Windows Event Logs.

Event Logs :

1. Application Error

Faulting application name: llserverworker.exe, version: 16.2.0.597, time stamp: 0x5bff5a06
Faulting module name: ucrtbase.DLL, version: 10.0.14393.2630, time stamp: 0x5bbec6c9
Exception code: 0xc0000409
Fault offset: 0x000000000006e83e
Faulting process id: 0x19d8
Faulting application start time: 0x01d4d317f262e059
Faulting application path: E:\OPENTEXT\bin\llserverworker.exe
Faulting module path: C:\Windows\SYSTEM32\ucrtbase.DLL
Report Id: 35a147f4-3f0b-11e9-80e3-000d3a61cf83
Faulting package full name:
Faulting package-relative application ID:

2. Event Error : COMRuntime

An unmarshaling policy check was performed when unmarshaling a custom marshaled object and the class {45FB4600-E6E8-4928-B25E-50476FF79425} was rejected

Regards,

Mayank Gupta

Hi Dave,

One more point to note here is that, when we edit any module, this error occurs and the oll file of that modules get corrupted (Reduced to 0 kb).

This happens with core as well as custom module.

Regards,

Mayank Gupta

Dave Carpeneto

Thanks for all the detail on things attempted. If I look @ the development & support tracking systems I don't see this anywhere prior to today, so it's odd that you're hitting this.

Is it possible for you to generate a dump file ( https://docs.microsoft.com/en-us/windows/desktop/wer/collecting-user-mode-dumps ) & attach it to the ticket? The exception code kind of implies an overflow.

Alternately (or in addition to): since the module complaining is shipped with the OS it'd be good to rule out badness in the OS. Is it possible for you to do the following? :

Run software update & ensure there's no items needing installed
Run the following to see if anything is flagged as corrupt @ the OS level:
- SFC /verifyonly
- Dism /Online /Cleanup-Image /ScanHealth
- Dism /Online /Cleanup-Image /CheckHealth

... the three commands listed above will flag (but not fix) anything seen. Should anything be reported: have a look about on the Microsoft websites for repair instructions.

Thanks, Dave

Dave Carpeneto

Quoted Mayank Gupta on 05/02/2019 04:04 PM:

One more point to note here is that, when we edit any module, this error occurs and the oll file of that modules get corrupted (Reduced to 0 kb).

Well that one is fixed in 16.2.8 (as per my comment in https://knowledge.opentext.com/go/75440658 earlier this morning :-) )

Mayank Gupta

Hi Dave,

I will work with our Infra team on System dumps. But I am not sure that this is due to system issues as we update system every week or so with latest updates from Microsoft. More over I have tried this on multiple machines and every where I was able to replicate the issue.

Due you know if any of the clients is successfully able to use CSIDE with 16.2.7 CS version? or do you have any other input which we can try. In the mean time we will check on the System Dumps.

Regards,

Mayank Gupta

Hany Samuel

Quoted Mayank Gupta on 05/03/2019 10:17 AM:

Hi Dave,

Due you know if any of the clients is successfully able to use CSIDE with 16.2.7 CS version? or do you have any other input which we can try. In the mean time we will check on the System Dumps.

Regards,

Mayank Gupta

Hi Mayank,

I took a look on the logs you attached to the ticket, it seems you work directly on the OLLs like the old builder way. I see you changed in ospaces like blog.oll and it failed to save the changes. In fact you will find blog.oll now truncated to zero bytes so you need to replace this oll from a backup you have or the content server will crash either with CSIDE or the service mode. This is a known bug introduced in content server 16.2.5 till 16.2.7 and we just fixed it in 16.2.8. a workaround to get you working with your current version of content server (after fixing the corrupted olls from a backup) is to import the source of any oll you plan to work on to your workspace and then restart the VM.

let me know how it goes with you

Mayank Gupta

Hi Hany,

Yes, We took the back of the modules.

I tried the workaround and it seems to work ok so far. Just wondering how it will behave if I orphan something from core module in my custom module. Or probably I need to import core module also before doing any work on them. Probably we will upgrade to 16.2.8 once we can. I will work on this further and see how it goes.

This was really helpful. Thanks for your guidance here.

Regards,

Mayank Gupta

Hany Samuel

Hi Mayank,

I am happy that things worked for you !

As for your question, whatever ospace you are going to change needs to be source imported before your change or you end up with the oll truncated. Long story short, the bug is simply lock/unlock takes no effect after the VM starts so we workaround that by importing the source to force whatever ospace to start in unlocked state so it can be modified.

if you orphan a feature then the parent ospace is not the one to change it is the destination ospace so just import that but if you are not sure just import both. Make sure that build automatically is turned off so the source build does not kick in, you do not want that to happen as you work directly on the memory layed oll which will be different than the imported source as you work.

all of this will not be needed as soon as you upgrade to 16.2.8.

Feel free to reach ou if you have any further questions !

Thanks,

Hany

Rashmi_S

hi Mayank,

We are also facing the same issue after patching CS to 16.2.5. Could you please help us here with any workaround that you did to edit the module or restore the module?

Appreciate your help.

Regards

Rashmi

Dave Carpeneto

Quoted Rashmi S on 05/19/2019 01:52 PM:

Could you please help us here with any workaround that you did to edit the module or restore the module?

Hi Rashmi - as per Hany's earlier note:

Quoted Rashmi S on 05/19/2019 01:52 PM:

[...] a workaround to get you working with your current version of content server (after fixing the corrupted olls from a backup) is to import the source of any oll you plan to work on to your workspace and then restart the VM.