Indexer does not finish likely because of Suggester hanging

[ChristopheBordieu]

Using OG 1.6.6 / Tomcat 10 / Java 11 on VM 32G RAM / 20 CPUs.
Indexer run OK:

2021-03-31 03:01:53.484+0000 INFO t1 Statistics.logIt: Done indexing data of all repositories (took 5:32:57)

and new configuration has started to be sent to Opengrok webapp:

2021-03-31 03:01:53.726+0000 INFO t1 Indexer.writeConfigToFile: Writing configuration to /srv/opengrok-var/etc/configuration.xml
2021-03-31 03:02:02.517+0000 INFO t1 Indexer.writeConfigToFile: Done...
2021-03-31 03:02:02.630+0000 INFO t1 Indexer.sendToConfigHost: Sending configuration to: http://localhost:8080/grok

but since then, indexer job (and ctags jobs) still up: no message has been reported to inform config has been successfully reloaded on webapp. Log opengrok0.0.log stuck at:

2021-03-31 03:02:10.326+0000 FINEST t1 ServiceFinder$LazyObjectIterator.next: Loading next object: org.glassfish.jersey.inject.hk2.Hk2InjectionManagerFactory
2021-03-31 03:02:10.327+0000 FINER t1 Logger.debug: Created ServiceLocator ServiceLocatorImpl(__HK2_Generated_1909,1909,1224021151)
2021-03-31 03:02:10.331+0000 FINE t1 AbstractHk2InjectionManager.<init>: Clearing Jersey HK2 caches. Service cache size: 3, reflection cache size: 20.
2021-03-31 03:02:10.344+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.logging.LoggingFeatureAutoDiscoverable
2021-03-31 03:02:10.345+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.internal.config.ExternalPropertiesAutoDiscoverable
2021-03-31 03:02:10.400+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.server.filter.internal.ServerFiltersAutoDiscoverable
2021-03-31 03:02:10.424+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.jackson.internal.JacksonAutoDiscoverable
2021-03-31 03:02:10.433+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.server.validation.internal.ValidationAutoDiscoverable
2021-03-31 03:02:10.434+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.server.wadl.internal.WadlAutoDiscoverable
2021-03-31 03:02:10.434+0000 FINEST t1 ServiceFinder$LazyClassIterator.next: Loading next class: org.glassfish.jersey.server.internal.monitoring.MonitoringAutodiscoverable
2021-03-31 03:02:10.435+0000 FINER t1 SystemPropertiesConfigurationModel.getProperties: System properties configuration provider not allowed
2021-03-31 03:02:10.437+0000 FINE t1 ClientExecutorProvidersConfigurator.lookupManagedScheduledExecutorService: null
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.glassfish.jersey.client.ClientExecutorProvidersConfigurator.lookupManagedScheduledExecutorService(ClientExecutorProvidersConfigurator.java:174)
        at org.glassfish.jersey.client.ClientExecutorProvidersConfigurator.init(ClientExecutorProvidersConfigurator.java:124)
        at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:452)
        at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:317)
        at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:814)
        at org.glassfish.jersey.client.ClientRequest.getClientRuntime(ClientRequest.java:176)
        at org.glassfish.jersey.client.ClientRequest.getInjectionManager(ClientRequest.java:567)
        at org.glassfish.jersey.client.JerseyWebTarget.onBuilder(JerseyWebTarget.java:364)
        at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:192)
        at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:36)
        at org.opengrok.indexer.configuration.RuntimeEnvironment.writeConfiguration(RuntimeEnvironment.java:1430)
        at org.opengrok.indexer.index.Indexer.sendToConfigHost(Indexer.java:1149)
        at org.opengrok.indexer.index.Indexer.main(Indexer.java:378)
Caused by: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or in an application resource file: java.naming.factory.initial
        at java.naming/javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:691)
        at java.naming/javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:305)
        at java.naming/javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:342)
        at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
        ... 16 more

2021-03-31 03:02:10.439+0000 CONFIG t1 ExtendedLogger.config: Selected ExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientAsyncExecutorProvider] to be used for injection of executor qualified by [org.glassfish.jersey.client.ClientAsyncExecutor] annotation.
2021-03-31 03:02:10.439+0000 CONFIG t1 ExtendedLogger.config: Selected ScheduledExecutorServiceProvider implementation [org.glassfish.jersey.client.DefaultClientBackgroundSchedulerProvider] to be used for injection of scheduler qualified by [org.glassfish.jersey.client.ClientBackgroundScheduler] annotation.
2021-03-31 03:02:10.452+0000 CONFIG t1 HttpUrlConnector.<init>: Restricted headers are not enabled using [sun.net.http.allowRestrictedHeaders] system property (setting only takes effect on connections created after the property has been set/changed).
2021-03-31 03:02:10.530+0000 FINEST t1 HttpURLConnection.plainConnect0: ProxySelector Request for http://localhost:8080/grok/api/v1/configuration?reindex=true
2021-03-31 03:02:10.608+0000 FINEST t1 HttpURLConnection.plainConnect0: Proxy used: DIRECT
2021-03-31 03:02:10.699+0000 FINE t1 HttpURLConnection.writeRequests: sun.net.www.MessageHeader@11718d927 pairs: {PUT /grok/api/v1/configuration?reindex=true HTTP/1.1: null}{Content-Type: application/xml}{User-Agent: Jersey/3.0.1 (HttpUrlConnection 11.0.9)}{Host: localhost:8080}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}{Content-Length: 27811969}

Attaching thread dumps of indexer and webapp JVMs. Looking quickly at them, it seems Suggester could be the issue.
File catalina.out shows many projects were rebuilt successfully. But no idea if the rebuild is over.

Webapp is not responding when searching: wheeling loop.

webapp.threaddump.txt.gz
indexer.threaddump.txt.gz

Could you help?
Is there a way to disable Suggester?

[ChristopheBordieu]

Is there a way to disable Suggester?
Yes: https://github.com/oracle/opengrok/wiki/Suggester. Thanks for this doc!

[vladak]

The configuration set API call handling is waiting on:

"http-nio-8080-exec-10" #38 daemon prio=5 os_prio=0 cpu=49903,96ms elapsed=63423,64s allocated=10424M defined_classes=380 tid=0x00007ff9ad14b800 nid=0x2bab1a waiting on condition  [0x00007ff89a1bf000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.9/Native Method)
        - parking to wait for  <0x000000021b3de0f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.9/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.9/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.9/AbstractQueuedSynchronizer.java:917)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.9/AbstractQueuedSynchronizer.java:1240)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.9/ReentrantReadWriteLock.java:959)
        at org.opengrok.suggest.SuggesterProjectData.close(SuggesterProjectData.java:485)
        at org.opengrok.suggest.Suggester.lambda$close$5(Suggester.java:611)
        at org.opengrok.suggest.Suggester$$Lambda$695/0x00000008006c9840.accept(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(java.base@11.0.9/ConcurrentHashMap.java:4772)
        at org.opengrok.suggest.Suggester.close(Suggester.java:609)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.refresh(SuggesterServiceImpl.java:167)
        at org.opengrok.web.api.v1.controller.ConfigurationController.set(ConfigurationController.java:63)

waiting on

opengrok/suggester/src/main/java/org/opengrok/suggest/SuggesterProjectData.java

Lines 484 to 485 in 5631d33

	public void close() throws IOException {
	lock.writeLock().lock();

which is RW lock that belongs to given SuggesterProjectData instance.

and there are suggester rebuild threads such as this one:

"ForkJoinPool.commonPool-worker-51" #30871 daemon prio=5 os_prio=0 cpu=34,47ms elapsed=35547,89s allocated=1034K defined_classes=16 tid=0x00007ff86c007800 nid=0x7165a waiting on condition  [0x00007ff8989aa000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.9/Native Method)
        - parking to wait for  <0x0000000180e711a8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.9/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.9/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.9/AbstractQueuedSynchronizer.java:1009)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.9/AbstractQueuedSynchronizer.java:1324)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.9/ReentrantReadWriteLock.java:738)
        at org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.rebuild(SuggesterServiceImpl.java:204)
        at org.opengrok.web.api.v1.controller.ProjectsController.lambda$markIndexed$3(ProjectsController.java:289)
        at org.opengrok.web.api.v1.controller.ProjectsController$$Lambda$1120/0x00000008007a7c40.run(Unknown Source)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(java.base@11.0.9/CompletableFuture.java:1736)
        at java.util.concurrent.CompletableFuture$AsyncRun.exec(java.base@11.0.9/CompletableFuture.java:1728)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@11.0.9/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@11.0.9/ForkJoinPool.java:1020)
        at java.util.concurrent.ForkJoinPool.scan(java.base@11.0.9/ForkJoinPool.java:1656)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.9/ForkJoinPool.java:1594)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.9/ForkJoinWorkerThread.java:183)

that is waiting for the Suggester wide RW lock (the read "side").

Also, there suggester threads that appear to be running, e.g.:

"ForkJoinPool-481-worker-19" #56983 daemon prio=5 os_prio=0 cpu=774856,97ms elapsed=27973,81s allocated=84237M defined_classes=0 tid=0x00007ff8f8181800 nid=0x7b556 runnable  [0x00007ff8994b4000]
   java.lang.Thread.State: RUNNABLE
        at net.openhft.chronicle.map.VanillaChronicleMap.get(VanillaChronicleMap.java:552)
        at java.util.concurrent.ConcurrentMap.getOrDefault(java.base@11.0.9/ConcurrentMap.java:88)
        at org.opengrok.suggest.popular.impl.chronicle.ChronicleMapAdapter.get(ChronicleMapAdapter.java:62)
        at org.opengrok.suggest.SuggesterProjectData.lambda$getSearchCounts$2(SuggesterProjectData.java:476)
        at org.opengrok.suggest.SuggesterProjectData$$Lambda$1122/0x00000008007e9440.get(Unknown Source)
        at org.opengrok.suggest.SuggesterProjectData$WFSTInputIterator.weight(SuggesterProjectData.java:603)
        at org.apache.lucene.search.suggest.SortedInputIterator.sort(SortedInputIterator.java:184)
        at org.apache.lucene.search.suggest.SortedInputIterator.<init>(SortedInputIterator.java:76)
        at org.apache.lucene.search.suggest.SortedInputIterator.<init>(SortedInputIterator.java:62)
        at org.apache.lucene.search.suggest.fst.WFSTCompletionLookup$WFSTInputIterator.<init>(WFSTCompletionLookup.java:273)
        at org.apache.lucene.search.suggest.fst.WFSTCompletionLookup.build(WFSTCompletionLookup.java:115)
        at org.opengrok.suggest.SuggesterProjectData.build(SuggesterProjectData.java:266)
        at org.opengrok.suggest.SuggesterProjectData.build(SuggesterProjectData.java:253)
        at org.opengrok.suggest.SuggesterProjectData.rebuild(SuggesterProjectData.java:238)
        at org.opengrok.suggest.Suggester.lambda$getRebuildRunnable$2(Suggester.java:347)
        at org.opengrok.suggest.Suggester$$Lambda$1121/0x00000008007a8840.run(Unknown Source)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(java.base@11.0.9/ForkJoinTask.java:1407)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@11.0.9/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@11.0.9/ForkJoinPool.java:1020)
        at java.util.concurrent.ForkJoinPool.scan(java.base@11.0.9/ForkJoinPool.java:1656)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.9/ForkJoinPool.java:1594)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.9/ForkJoinWorkerThread.java:183)

So it may be that once the running suggester threads finish, everything will unblock.

It would make sense to untangle the lock dependencies and see if this is a deadlock or not.

Reminds me of #3468. This might be a duplicate of that one.

[ChristopheBordieu]

Has the property minChars any impact regarding the time needed by these Suggester threads to finish when rebuilding data? Or not, it only impacts usage when typing a value into field?

So, according to you, If I had let the job runs, eventually it would have finish.
I killed it restarting indexer with Suggester disabled for the moment just to be sure loading of new config is done successfully.
I will then eventually rerun indexer with Suggester enabled.

[ahornace]

Has the property minChars any impact regarding the time needed by these Suggester threads to finish when rebuilding data?

No, since that would have very minimal or none at all influence on the size of the data structure. However, if we were to take it into consideration, it could theoretically slightly improve the performance of building the data structure but I doubt it would be noticeable.

Or not, it only impacts usage when typing a value into field?

Yes.

[ChristopheBordieu]

• Disabled Suggester, run indexer: indexing done, config loaded OK, quick test on webapp are OK.
• Enabled Suggester, set minChars to 3, rerun indexer (no code source updates): indexing done, config loaded OK, quick test on webapp OK (suggestions are provided as soon as 3 chars are typed).

Don't think these results help you, and not totally me...

[ChristopheBordieu]

I am still facing the same issue with a new indexer job. I let the job run more than 1 day but it did not finish.
I am wondering if, with large number of repositories with long histories, disabling some fields in Suggeste could help, in particular path field?

[ahornace]

I am wondering if, with large number of repositories with long histories, disabling some fields in Suggeste could help, in particular path field?

This should help a little. Suggester tries to (re)build for each field respectively. For example, suggester will only store and (re)build 4 data structures instead of 5 for each project if you leave out one field.

However, there is still something fishy and probably there is a couple of bugs in there. I'll try to come up with some optimizations for the suggester.

[ChristopheBordieu]

Gonna try.
Is there a way to increase logs from Suggester on Tomcat size?

[ChristopheBordieu]

How do I set property allowedFields which is a set of strings?
<string>... ? <setString>... ?

[ahornace]

  <void id="SuggesterConfig0" property="suggesterConfig">
   <void property="allowedFields">
    <void method="clear"/>
    <void method="add">
     <string>defs</string>
    </void>
    <void method="add">
     <string>path</string>
    </void>
    <void method="add">
     <string>hist</string>
    </void>
    <void method="add">
     <string>refs</string>
    </void>
    <void method="add">
     <string>type</string>
    </void>
    <void method="add">
     <string>full</string>
    </void>
   </void>
  </void>

I believe you can just remove this part:

<void method="add">
     <string>path</string>
</void>

[ChristopheBordieu]

HI @ahornace

Thanks. But it fails. Here is the stack.

==> catalina.out <==
06-Apr-2021 07:29:33.445 WARNING [main] org.opengrok.web.WebappListener.contextInitialized Configuration error. Failed to read config file: 
	java.io.IOException: java.lang.UnsupportedOperationException
		at org.opengrok.indexer.configuration.Configuration.decodeObject(Configuration.java:1441)
		at org.opengrok.indexer.configuration.Configuration.read(Configuration.java:1406)
		at org.opengrok.indexer.configuration.RuntimeEnvironment.readConfiguration(RuntimeEnvironment.java:1394)
		at org.opengrok.web.WebappListener.contextInitialized(WebappListener.java:84)
		at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4665)
		at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5126)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
		at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
		at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:743)
		at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1026)
		at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1993)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
		at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
		at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:827)
		at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:477)
		at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1701)
		at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:320)
		at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
		at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
		at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
		at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936)
		at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:880)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
		at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
		at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
		at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
		at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.StandardService.startInternal(StandardService.java:434)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.startup.Catalina.start(Catalina.java:795)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
		at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
	Caused by: java.lang.UnsupportedOperationException
		at java.base/java.util.Collections$UnmodifiableCollection.clear(Collections.java:1079)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
		at jdk.internal.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260)
		at java.desktop/java.beans.Statement.invokeInternal(Statement.java:304)
		at java.desktop/java.beans.Statement$2.run(Statement.java:187)
		at java.base/java.security.AccessController.doPrivileged(Native Method)
		at java.desktop/java.beans.Statement.invoke(Statement.java:184)
		at java.desktop/java.beans.Expression.getValue(Expression.java:155)
		at java.desktop/com.sun.beans.decoder.ObjectElementHandler.getValueObject(ObjectElementHandler.java:166)
		at java.desktop/com.sun.beans.decoder.NewElementHandler.getValueObject(NewElementHandler.java:123)
		at java.desktop/com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169)
		at java.desktop/com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:320)
		at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:610)
		at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:184)
		at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1377)
		at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2710)
		at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
		at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534)
		at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
		at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
		at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
		at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
		at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
		at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
		at java.desktop/com.sun.beans.decoder.DocumentHandler$1.run(DocumentHandler.java:377)
		at java.desktop/com.sun.beans.decoder.DocumentHandler$1.run(DocumentHandler.java:374)
		at java.base/java.security.AccessController.doPrivileged(Native Method)
		at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
		at java.desktop/com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:374)
		at java.desktop/java.beans.XMLDecoder$1.run(XMLDecoder.java:201)
		at java.desktop/java.beans.XMLDecoder$1.run(XMLDecoder.java:199)
		at java.base/java.security.AccessController.doPrivileged(Native Method)
		at java.desktop/java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:199)
		at java.desktop/java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
		at org.opengrok.indexer.configuration.Configuration.decodeObject(Configuration.java:1428)
		... 44 more