SOLR crash during unwanted numeric -> long conversion

[Laurira]

Description:

I have a siard from institution that has very different data types in it. According to the log file of siard creation there are no errors. Also this siard can be loaded to DBPTK easily.
Now if "Browse" is clicked the DBPTK starts to index 300MB siard file with SOLR. It tried to process it about 72 hours and then everything crashed (error was about running out of disk space).

Log file dbvtk.log tells that:
2021-02-23 10:59:03,791 [http-nio-auto-1-exec-5] ERROR o.a.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: ERROR: [doc=53132] Error adding field 'col10_l'='6522022.9' msg=For input string: "6522022.9"
....
Caused by: java.lang.NumberFormatException: For input string: "6522022.9"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.solr.schema.LongPointField.createField(LongPointField.java:154)
at org.apache.solr.schema.PointField.createFields(PointField.java:250)
at org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:65)
at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:171)
... 130 more

So it tried to convert the value "'6522022.9" to LONG which is integer and can not be done.

Question is why does DBPTK think that these values have to be converted into LONG?

As the value "6522022.9" is quite uniq then it was possible to locate it in the table89.xml that has following metadata:

                <table>
                    <name>observations_aud</name>
                    <folder>table89</folder>
                    <description>Vaatlused</description>
                    <columns>
                        <column>
                            <name>rev</name>
                            <type>INTEGER</type>
                            <typeOriginal>int4</typeOriginal>
                            <nullable>false</nullable>
                        </column>
 ...
                        <column>
                            <name>latitude</name>
                            <type>NUMERIC</type>
                            <typeOriginal>numeric</typeOriginal>
                            <nullable>true</nullable>
                            <description>X-koordinaat</description>
                        </column>
...

So the problematic column is "latitude" and type should be NUMERIC.

Why does DBPTK try to convert numeric values to LONG during SOLR indexing?

[hmiguim]

Hi Lauri,

Check the screenshots bellow. I have a table with a column NUMERIC(10,2) data type. Successfully imported to DBPTK Enterprise and the value have a decimal point also.

Could you please provide the DBMS used to produce this SIARD?

[Laurira]

Hi Miguel,

I sent you an e-mail with extra information. I can not share it to everyone.

Thanks,
Lauri

[hmiguim]

DBPTK-UI is behaving as excepted, the problem lies with DBPTK Developer follow keeps/dbptk-developer#477