org.hibernate.search.exception.SearchException: HSEARCH000083: Unable to serialize List<LuceneWork>

Hibernate JIRA | Steffen Terheiden | 5 months ago
  1. 0

    While setting up the JMS replication for the Lucene index via Hibernate Search i came across an error that complains about *TokenStream contract violation* (see the stack trace at the end of the description). After a research in the web i found out, that this is usually caused by the update of the TokenStream API that now requires a defined workflow to use it ( https://lucene.apache.org/core/5_3_0/core/org/apache/lucene/analysis/TokenStream.html ). After investigating all possible problems in my application, i reviewed the source code of the serialization classes in Hibernate Search. So i discovered that you use the TokenStream in your class _org.hibernate.search.indexes.serialization.impl.CopyTokenStream_ but don't comply with the defined workflow. In your method _createAttributeLists_ you need to insert a short line to reset the "input"-TokenStream. Testing my idea i added the method call and all works as expected. Below my code of the createAttributeList method in org.hibernate.search.indexes.serialization.impl.CopyTokenStream: {code:java} private static List<List<AttributeImpl>> createAttributeLists(TokenStream input) throws IOException { List<List<AttributeImpl>> results = new ArrayList<>(); // added input.reset(), see API TokenStream input.reset(); while ( input.incrementToken() ) { List<AttributeImpl> attrs = new ArrayList<>(); results.add( attrs ); Iterator<AttributeImpl> iter = input.getAttributeImplsIterator(); while ( iter.hasNext() ) { //we need to clone as AttributeImpl instances can be reused across incrementToken() calls attrs.add( iter.next().clone() ); } } input.end(); return results; } {code} Is there any hidden purpose that you don't use the reset() call or is it just a bug? *Below the mentioned stack trace:* org.hibernate.search.exception.SearchException: HSEARCH000083: Unable to serialize List<LuceneWork> at org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.toSerializedModel(LuceneWorkSerializerImpl.java:109) at org.hibernate.search.backend.jms.impl.JmsBackendQueueTask.run(JmsBackendQueueTask.java:61) at org.hibernate.search.backend.jms.impl.JmsBackendQueueProcessor.applyWork(JmsBackendQueueProcessor.java:88) at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:112) at org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:49) at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:81) at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.flushWorks(PostTransactionWorkQueueSynchronization.java:114) at org.hibernate.search.backend.impl.TransactionalWorker.flushWorks(TransactionalWorker.java:165) at org.hibernate.search.impl.FullTextSessionImpl.flushToIndexes(FullTextSessionImpl.java:87) at com.sobis.jaf.JAFApplication.createIndexFor(JAFApplication.java:919) at com.sobis.jaf.JAFApplication.createIndexAndVerify(JAFApplication.java:820) at com.sobis.jaf.JAFApplication.createIndex(JAFApplication.java:796) at com.sobis.jaf.JAFApplication.createIndex(JAFApplication.java:672) at com.sobis.jaf.JAFApplication$1.performAction(JAFApplication.java:486) at com.sobis.jaf.services.thread.JAFThread.run(JAFThread.java:71) Caused by: java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow. at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:111) at org.apache.lucene.analysis.core.KeywordTokenizer.incrementToken(KeywordTokenizer.java:68) at org.hibernate.search.indexes.serialization.impl.CopyTokenStream.createAttributeLists(CopyTokenStream.java:85) at org.hibernate.search.indexes.serialization.impl.CopyTokenStream.buildSerializableTokenStream(CopyTokenStream.java:39) at org.hibernate.search.indexes.serialization.spi.LuceneFieldContext.getTokenStream(LuceneFieldContext.java:137) at org.hibernate.search.indexes.serialization.avro.impl.AvroSerializer.addFieldWithTokenStreamData(AvroSerializer.java:281) at org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.serializeField(LuceneWorkSerializerImpl.java:237) at org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.serializeDocument(LuceneWorkSerializerImpl.java:175) at org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.toSerializedModel(LuceneWorkSerializerImpl.java:97) ... 14 more

    Hibernate JIRA | 5 months ago | Steffen Terheiden
    org.hibernate.search.exception.SearchException: HSEARCH000083: Unable to serialize List<LuceneWork>
  2. 0

    Why am I getting an IllegalStateException when trying to add a document that has a Textfield in lucene

    Stack Overflow | 2 years ago | John Smith
    java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
  3. 0

    Michele Rizzi Web Page – Lucene:information retrival

    rizzimichele.it | 1 year ago
    java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

  5. 0

    IKAnalyzer2012FF + Lucene4.9 TokenStream contract violation: reset()/close() call missing - qiruiduni的专栏 - 博客频道 - CSDN.NET

    csdn.net | 1 year ago
    java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
  6. 0

    Error in Keyword extraction using lucene

    Stack Overflow | 2 years ago | user3774248
    java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.

    1 unregistered visitors
    Not finding the right solution?
    Take a tour to get the most out of Samebug.

    Tired of useless tips?

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. java.lang.IllegalStateException

      TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.

      at org.apache.lucene.analysis.Tokenizer$1.read()
    2. Lucene
      Tokenizer$1.read
      1. org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:111)
      1 frame
    3. Lucene Common Analyzers
      KeywordTokenizer.incrementToken
      1. org.apache.lucene.analysis.core.KeywordTokenizer.incrementToken(KeywordTokenizer.java:68)
      1 frame
    4. Hibernate Search Engine
      LuceneWorkSerializerImpl.toSerializedModel
      1. org.hibernate.search.indexes.serialization.impl.CopyTokenStream.createAttributeLists(CopyTokenStream.java:85)
      2. org.hibernate.search.indexes.serialization.impl.CopyTokenStream.buildSerializableTokenStream(CopyTokenStream.java:39)
      3. org.hibernate.search.indexes.serialization.spi.LuceneFieldContext.getTokenStream(LuceneFieldContext.java:137)
      4. org.hibernate.search.indexes.serialization.avro.impl.AvroSerializer.addFieldWithTokenStreamData(AvroSerializer.java:281)
      5. org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.serializeField(LuceneWorkSerializerImpl.java:237)
      6. org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.serializeDocument(LuceneWorkSerializerImpl.java:175)
      7. org.hibernate.search.indexes.serialization.impl.LuceneWorkSerializerImpl.toSerializedModel(LuceneWorkSerializerImpl.java:97)
      7 frames
    5. org.hibernate.search
      JmsBackendQueueProcessor.applyWork
      1. org.hibernate.search.backend.jms.impl.JmsBackendQueueTask.run(JmsBackendQueueTask.java:61)
      2. org.hibernate.search.backend.jms.impl.JmsBackendQueueProcessor.applyWork(JmsBackendQueueProcessor.java:88)
      2 frames
    6. Hibernate Search Engine
      FullTextSessionImpl.flushToIndexes
      1. org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:112)
      2. org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:49)
      3. org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:81)
      4. org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.flushWorks(PostTransactionWorkQueueSynchronization.java:114)
      5. org.hibernate.search.backend.impl.TransactionalWorker.flushWorks(TransactionalWorker.java:165)
      6. org.hibernate.search.impl.FullTextSessionImpl.flushToIndexes(FullTextSessionImpl.java:87)
      6 frames
    7. com.sobis.jaf
      JAFThread.run
      1. com.sobis.jaf.JAFApplication.createIndexFor(JAFApplication.java:919)
      2. com.sobis.jaf.JAFApplication.createIndexAndVerify(JAFApplication.java:820)
      3. com.sobis.jaf.JAFApplication.createIndex(JAFApplication.java:796)
      4. com.sobis.jaf.JAFApplication.createIndex(JAFApplication.java:672)
      5. com.sobis.jaf.JAFApplication$1.performAction(JAFApplication.java:486)
      6. com.sobis.jaf.services.thread.JAFThread.run(JAFThread.java:71)
      6 frames