org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'keys' for entity 'entity[Participant:4719944893]'

JIRA | Philippe Laflamme | 7 years ago
  1. 0

    Running concurrent imports is possible by submitting multiple import jobs. The problem is with concurrent participant keys insert and lookup. Here is the sequence: T1: Thread 1 starts importing (and creating participant keys) T2: Thread 2 starts importing (and creating participant keys) T3: Thread 1 commits its data (with keys) T4: Thread 2 reads the ValueSet for an entity that Thread 1 created during its import: NoSuchValueSetException is thrown The trace may differ slightly, but will happen when Thread 2 tries to iterate on the value sets in the keys table. The ValueTable is in an inconsistent state: it returns a VariableEntity for which it cannot return a ValueSet. org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'keys' for entity 'entity[Participant:4719944893]' at org.obiba.magma.datasource.hibernate.HibernateValueTable.getValueSet(HibernateValueTable.java:75) at org.obiba.magma.support.AbstractValueTable$1.apply(AbstractValueTable.java:74) at org.obiba.magma.support.AbstractValueTable$1.apply(AbstractValueTable.java:71) at com.google.common.collect.Iterators$8.next(Iterators.java:697) at org.obiba.opal.core.service.impl.OpalPrivateVariableEntityMap.publicEntity(OpalPrivateVariableEntityMap.java:69) at org.obiba.opal.core.magma.PrivateVariableEntityValueTable$1.apply(PrivateVariableEntityValueTable.java:75) at org.obiba.opal.core.magma.PrivateVariableEntityValueTable$1.apply(PrivateVariableEntityValueTable.java:1) at org.obiba.magma.views.ValueSetWrapper.getVariableEntity(ValueSetWrapper.java:27) at org.obiba.magma.support.DatasourceCopier$LoggingListener.onValueSetCopy(DatasourceCopier.java:352) at org.obiba.magma.support.DatasourceCopier.notifyListeners(DatasourceCopier.java:273) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:200) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:180) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:160) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:152) at org.obiba.opal.core.service.impl.DefaultImportService.copyPublicViewToDestinationDatasource(DefaultImportService.java:193) at org.obiba.opal.core.service.impl.DefaultImportService.copyParticipants(DefaultImportService.java:154) at org.obiba.opal.core.service.impl.DefaultImportService.copyValueTables(DefaultImportService.java:134) at org.obiba.opal.core.service.impl.DefaultImportService.copyToDestinationDatasource(DefaultImportService.java:125) at org.obiba.opal.core.service.impl.DefaultImportService.importData(DefaultImportService.java:118) at org.obiba.opal.core.service.impl.DefaultImportService.importData(DefaultImportService.java:100) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy44.importData(Unknown Source) at org.obiba.opal.shell.commands.ImportCommand.importFiles(ImportCommand.java:95) at org.obiba.opal.shell.commands.ImportCommand.execute(ImportCommand.java:52) at org.obiba.opal.shell.CommandJob.run(CommandJob.java:109) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

    JIRA | 7 years ago | Philippe Laflamme
    org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'keys' for entity 'entity[Participant:4719944893]'
  2. 0

    Running concurrent imports is possible by submitting multiple import jobs. The problem is with concurrent participant keys insert and lookup. Here is the sequence: T1: Thread 1 starts importing (and creating participant keys) T2: Thread 2 starts importing (and creating participant keys) T3: Thread 1 commits its data (with keys) T4: Thread 2 reads the ValueSet for an entity that Thread 1 created during its import: NoSuchValueSetException is thrown The trace may differ slightly, but will happen when Thread 2 tries to iterate on the value sets in the keys table. The ValueTable is in an inconsistent state: it returns a VariableEntity for which it cannot return a ValueSet. org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'keys' for entity 'entity[Participant:4719944893]' at org.obiba.magma.datasource.hibernate.HibernateValueTable.getValueSet(HibernateValueTable.java:75) at org.obiba.magma.support.AbstractValueTable$1.apply(AbstractValueTable.java:74) at org.obiba.magma.support.AbstractValueTable$1.apply(AbstractValueTable.java:71) at com.google.common.collect.Iterators$8.next(Iterators.java:697) at org.obiba.opal.core.service.impl.OpalPrivateVariableEntityMap.publicEntity(OpalPrivateVariableEntityMap.java:69) at org.obiba.opal.core.magma.PrivateVariableEntityValueTable$1.apply(PrivateVariableEntityValueTable.java:75) at org.obiba.opal.core.magma.PrivateVariableEntityValueTable$1.apply(PrivateVariableEntityValueTable.java:1) at org.obiba.magma.views.ValueSetWrapper.getVariableEntity(ValueSetWrapper.java:27) at org.obiba.magma.support.DatasourceCopier$LoggingListener.onValueSetCopy(DatasourceCopier.java:352) at org.obiba.magma.support.DatasourceCopier.notifyListeners(DatasourceCopier.java:273) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:200) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:180) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:160) at org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:152) at org.obiba.opal.core.service.impl.DefaultImportService.copyPublicViewToDestinationDatasource(DefaultImportService.java:193) at org.obiba.opal.core.service.impl.DefaultImportService.copyParticipants(DefaultImportService.java:154) at org.obiba.opal.core.service.impl.DefaultImportService.copyValueTables(DefaultImportService.java:134) at org.obiba.opal.core.service.impl.DefaultImportService.copyToDestinationDatasource(DefaultImportService.java:125) at org.obiba.opal.core.service.impl.DefaultImportService.importData(DefaultImportService.java:118) at org.obiba.opal.core.service.impl.DefaultImportService.importData(DefaultImportService.java:100) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy44.importData(Unknown Source) at org.obiba.opal.shell.commands.ImportCommand.importFiles(ImportCommand.java:95) at org.obiba.opal.shell.commands.ImportCommand.execute(ImportCommand.java:52) at org.obiba.opal.shell.CommandJob.run(CommandJob.java:109) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

    JIRA | 7 years ago | Philippe Laflamme
    org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'keys' for entity 'entity[Participant:4719944893]'
  3. 0

    Running the following command the first time succeeds: administrator@opal> copy -o ankle-data.csv -v foo.AnkleBrachial Running the command again results in the following error: administrator@opal> copy -o ankle-data.csv -v foo.AnkleBrachial No ValueSet in table 'AnkleBrachial' for entity 'entity[Participant:9213340743]' This is also true for excel and fs: administrator@opal> copy -o ankle-data.xlsx -v foo.AnkleBrachial administrator@opal> copy -o ankle-data.zip -v foo.AnkleBrachial The error log contains: FINEST|6282/0|10-07-20 09:51:23|org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'AnkleBrachial' for entity 'entity[Participant:9213340743]' INFO|wrapper|10-07-20 09:51:23|Trigger found: Exception FINEST|6282/0|10-07-20 09:51:23| at org.obiba.magma.views.View.getValueSet(View.java:144) FINEST|6282/0|10-07-20 09:51:23| at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetReader.run(MultithreadedDatasourceCopier.java:222) FINEST|6282/0|10-07-20 09:51:23| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) FINEST|6282/0|10-07-20 09:51:23| at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) FINEST|6282/0|10-07-20 09:51:23| at java.util.concurrent.FutureTask.run(FutureTask.java:166) INFO|wrapper|10-07-20 09:51:23|start script scripts/trayMessage.gv FINEST|6282/0|10-07-20 09:51:23| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) FINEST|6282/0|10-07-20 09:51:23| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) INFO|wrapper|10-07-20 09:51:23|end script scripts/trayMessage.gv FINEST|6282/0|10-07-20 09:51:23| at org.obiba.opal.core.service.impl.DefaultExportServiceImpl$TransactionalThread$1.doInTransactionWithoutResult(DefaultExportServiceImpl.java:258) FINEST|6282/0|10-07-20 09:51:23| at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33) FINEST|6282/0|10-07-20 09:51:23| at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130) FINEST|6282/0|10-07-20 09:51:23| at org.obiba.opal.core.service.impl.DefaultExportServiceImpl$TransactionalThread.run(DefaultExportServiceImpl.java:255) The resulting ankle-data.csv file contains the ValueSet claimed to be missing: "entity_id","InstrumentRun.Contraindication.code","InstrumentRun.Contraindication.type","InstrumentRun.instrumentBarcode","InstrumentRun.otherContraindication","InstrumentRun.timeEnd","InstrumentRun.timeStart","InstrumentRun.user","IV_ABI_ARM_SUGG","IV_ABI_ARM_SUGG.captureMethod","IV_ABI_ARM_USED","IV_ABI_ARM_USED.captureMethod","IV_ABI_LEG_SUGG","IV_ABI_LEG_SUGG.captureMethod","IV_ABI_LEG_USED","IV_ABI_LEG_USED.captureMethod","RES_BRACHIAL_BP","RES_BRACHIAL_BP.captureMethod","RES_DORSALIS_PEDIS_BP","RES_DORSALIS_PEDIS_BP.captureMethod","RES_POSTERIOR_TIBIAL_BP","RES_POSTERIOR_TIBIAL_BP.captureMethod" "9213340743",,,"120002",,"2009-12-16T17:46:09.000-0500","2009-12-16T17:43:25.000-0500","administrator","LEFT_ARM_ABI","AUTOMATIC","LEFT","MANUAL","LEFT_LEG","AUTOMATIC","LEFT","MANUAL","150","MANUAL","160","MANUAL","170","MANUAL"

    JIRA | 6 years ago | Tony DeBat
    org.obiba.magma.NoSuchValueSetException: No ValueSet in table 'AnkleBrachial' for entity 'entity[Participant:9213340743]' INFO|wrapper|10-07-20 09:51:23|Trigger found: Exception
  4. Speed up your debug routine!

    Automated exception search integrated into your IDE

    Root Cause Analysis

    1. org.obiba.magma.NoSuchValueSetException

      No ValueSet in table 'keys' for entity 'entity[Participant:4719944893]'

      at org.obiba.magma.datasource.hibernate.HibernateValueTable.getValueSet()
    2. org.obiba.magma
      AbstractValueTable$1.apply
      1. org.obiba.magma.datasource.hibernate.HibernateValueTable.getValueSet(HibernateValueTable.java:75)
      2. org.obiba.magma.support.AbstractValueTable$1.apply(AbstractValueTable.java:74)
      3. org.obiba.magma.support.AbstractValueTable$1.apply(AbstractValueTable.java:71)
      3 frames
    3. Guava
      Iterators$8.next
      1. com.google.common.collect.Iterators$8.next(Iterators.java:697)
      1 frame
    4. org.obiba.opal
      PrivateVariableEntityValueTable$1.apply
      1. org.obiba.opal.core.service.impl.OpalPrivateVariableEntityMap.publicEntity(OpalPrivateVariableEntityMap.java:69)
      2. org.obiba.opal.core.magma.PrivateVariableEntityValueTable$1.apply(PrivateVariableEntityValueTable.java:75)
      3. org.obiba.opal.core.magma.PrivateVariableEntityValueTable$1.apply(PrivateVariableEntityValueTable.java:1)
      3 frames
    5. org.obiba.magma
      DatasourceCopier.copy
      1. org.obiba.magma.views.ValueSetWrapper.getVariableEntity(ValueSetWrapper.java:27)
      2. org.obiba.magma.support.DatasourceCopier$LoggingListener.onValueSetCopy(DatasourceCopier.java:352)
      3. org.obiba.magma.support.DatasourceCopier.notifyListeners(DatasourceCopier.java:273)
      4. org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:200)
      5. org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:180)
      6. org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:160)
      7. org.obiba.magma.support.DatasourceCopier.copy(DatasourceCopier.java:152)
      7 frames
    6. org.obiba.opal
      DefaultImportService.importData
      1. org.obiba.opal.core.service.impl.DefaultImportService.copyPublicViewToDestinationDatasource(DefaultImportService.java:193)
      2. org.obiba.opal.core.service.impl.DefaultImportService.copyParticipants(DefaultImportService.java:154)
      3. org.obiba.opal.core.service.impl.DefaultImportService.copyValueTables(DefaultImportService.java:134)
      4. org.obiba.opal.core.service.impl.DefaultImportService.copyToDestinationDatasource(DefaultImportService.java:125)
      5. org.obiba.opal.core.service.impl.DefaultImportService.importData(DefaultImportService.java:118)
      6. org.obiba.opal.core.service.impl.DefaultImportService.importData(DefaultImportService.java:100)
      6 frames
    7. Java RT
      Method.invoke
      1. sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2. sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      3. sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      4. java.lang.reflect.Method.invoke(Method.java:597)
      4 frames
    8. Spring AOP
      ReflectiveMethodInvocation.proceed
      1. org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
      2. org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      3. org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      3 frames
    9. Spring Tx
      TransactionInterceptor.invoke
      1. org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
      1 frame
    10. Spring AOP
      JdkDynamicAopProxy.invoke
      1. org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      2. org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      2 frames
    11. Unknown
      $Proxy44.importData
      1. $Proxy44.importData(Unknown Source)
      1 frame
    12. org.obiba.opal
      CommandJob.run
      1. org.obiba.opal.shell.commands.ImportCommand.importFiles(ImportCommand.java:95)
      2. org.obiba.opal.shell.commands.ImportCommand.execute(ImportCommand.java:52)
      3. org.obiba.opal.shell.CommandJob.run(CommandJob.java:109)
      3 frames
    13. Java RT
      Thread.run
      1. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      2. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      3. java.lang.Thread.run(Thread.java:619)
      3 frames